Distributed load balancer

ABSTRACT

A distributed load balancer in which a router receives packets from at least one client and routes packet flows to multiple load balancer (LB) nodes according to a per-flow hashed multipath routing technique. For a given packet flow, the LB nodes randomly select a server node as a target for the packet flow from among multiple server nodes and send a connection request to the server node. A load balancer module on the server node makes the decision on whether to accept or reject the connection based on one or more metrics indicating a respective server&#39;s current load. If the module accepts the connection request, a connection is established between the server and the client. Otherwise, the load balancer nodes may select another server node and try again. The connections established between clients and servers pass through the load balancer nodes, but are not terminated at the load balancer nodes.

BACKGROUND

Conventional load balancers are typically single, dedicated boxes thatinclude multiple network interface controllers (NICs), for example eightNICs, with some of the NICs handling inbound traffic from/outboundtraffic to clients and the other NICs handling outbound trafficfrom/inbound traffic to the host devices (e.g., servers such as webservers) that are being load balanced. Bandwidth or throughput on theseconventional load balancers is typically in the range of 40 Gigabits persecond (Gbps) on the client side and 40 Gbps on the server side. As thescale and scope of network-based applications and network-based servicessuch as cloud computing services have increased, data centers may househundreds or even thousands of host devices (e.g., web servers) that needto be load balanced. Conventional load balancers may not scale well insuch environments.

Further, conventional load balancers typically use techniques such asmax connections (or max conns), round robin, and/or least connections(least conns) applied to data collected from the host devices to selectwhich host device will handle a connection. In addition, conventionalload balancers typically serve as proxies to the host devices that theyfront and thus terminate connections (e.g., Transmission ControlProtocol (TCP) connections) from the clients and send the client trafficto the host devices on TCP connections established between the hostdevices and the load balancer. Thus, a host device and a client do notcommunicate over a direct TCP connection when using these conventionalload balancers.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example distributed load balancingsystem, according to at least some embodiments.

FIG. 2 is a high-level flowchart of a load balancing method that may beimplemented by the distributed load balancer system of FIG. 1, accordingto at least some embodiments.

FIG. 3 shows an example load balancer node that includes ingress,egress, and flow tracker components, according to at least someembodiments.

FIG. 4 illustrates routing and packet flow in the distributed loadbalancer, according to at least some embodiments.

FIG. 5 illustrates advertising ingress nodes to the edge router,according to at least some embodiments.

FIG. 6 is a flowchart of a multipath routing method, according to atleast some embodiments.

FIG. 7 graphically illustrates asymmetric packet flow, according to atleast some embodiments.

FIG. 8 illustrates packet flow in the distributed load balancing system,according to at least some embodiments.

FIGS. 9A and 9B provide a flowchart of packet flow when establishingconnections in the distributed load balancing system, according to atleast some embodiments.

FIGS. 10A through 10G illustrate packet flow in the distributed loadbalancing system, according to at least some embodiments.

FIGS. 11A through 11D illustrate the handling of events that effectmembership in the load balancer node consistent hash ring, according toat least some embodiments.

FIG. 12 is a high-level flowchart of a health check method that may beperformed by each load balancer node according to a health checkinterval, according to at least some embodiments.

FIG. 13 illustrates a method for health checking a load balancer nodefrom another load balancer node, according to at least some embodiments.

FIG. 14 graphically illustrates a load balancer node health checking oneor more other load balancer nodes, according to at least someembodiments.

FIG. 15 illustrates the load balancer nodes health checking the servernodes, according to at least some embodiments.

FIG. 16 graphically illustrates a view of health of another node thatmay be maintained by a load balancer node 110, according to at leastsome embodiments.

FIG. 17 illustrates health information that may be maintained by eachload balancer node, according to at least some embodiments.

FIGS. 18A and 18B illustrate handling a load balancer node failure,according to at least some embodiments.

FIGS. 19A and 19B graphically illustrate a connection publishingtechnique, according to at least some embodiments.

FIG. 20 is a high-level flowchart of a connection publishing method thatmay be performed by each load balancer module, according to at leastsome embodiments.

FIG. 21 is a flowchart of a method for distributing the activeconnection information received in a connection publishing packet totarget load balancer nodes, according to at least some embodiments.

FIG. 22 illustrates an alternative method for distributing the activeconnection information received in a connection publishing packet totarget load balancer nodes, according to at least some embodiments.

FIG. 23 illustrates example software stack architecture for a loadbalancer node according to at least some embodiments.

FIG. 24 illustrates aspects of core packet processing technology thatmay be used in embodiments.

FIG. 25 illustrates an example multicore packet processor for processingdata flows on the load balancer nodes, according to at least someembodiments.

FIG. 26 illustrates another example multicore packet processor forprocessing data flows on the load balancer nodes, according to at leastsome embodiments.

FIG. 27 illustrates processing of incoming packets by a load balancernode process, according to at least some embodiments.

FIG. 28 illustrates processing of outgoing packets by a load balancernode process, according to at least some embodiments.

FIG. 29 illustrates a load balancing system that includes a distributedload balancer in a production environment, according to at least someembodiments.

FIG. 30 illustrates a distributed load balancer test system thatincorporates a message bus mechanism that enables multiple distributedload balancing system components to be configured and executed in or asa single process, according to at least some embodiments.

FIGS. 31 and 32 illustrate message bus packet adapters and packetpipelines, according to at least some embodiments.

FIG. 33A illustrates an example provider network environment, accordingto at least some embodiments.

FIG. 33B illustrates a distributed load balancer implementation in anexample provider network environment as shown in FIG. 33A, according toat least some embodiments.

FIG. 34A illustrates an example physical rack implementation of thedistributed load balancer and server nodes according to at least someembodiments.

FIG. 34B illustrates another example physical rack implementation of thedistributed load balancer and server nodes according to at least someembodiments.

FIG. 35 illustrates an example networking environment in which one, twoor more distributed load balancers are implemented in a network,according to at least some embodiments.

FIG. 36 is a block diagram illustrating an example computer system thatmay be used in some embodiments.

While embodiments are described herein by way of example for severalembodiments and illustrative drawings, those skilled in the art willrecognize that embodiments are not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit embodiments tothe particular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope as defined by the appended claims. The headings usedherein are for organizational purposes only and are not meant to be usedto limit the scope of the description or the claims. As used throughoutthis application, the word “may” is used in a permissive sense (i.e.,meaning having the potential to), rather than the mandatory sense (i.e.,meaning must). Similarly, the words “include”, “including”, and“includes” mean including, but not limited to.

DETAILED DESCRIPTION

Various embodiments of methods and systems for distributed loadbalancing in network environments are described. Embodiments of adistributed load balancing method and system are described that may beimplemented according to embodiments of a distributed load balancer invarious network environments. Embodiments of the distributed loadbalancer may, for example, be used to facilitate and maintain packetflows, for example Transmission Control Protocol (TCP) technology packetflows, between clients on an external network such as the Internet anddestinations, typically servers (e.g., web servers, application servers,data servers, etc.) on a local network, such as a provider network 1900as illustrated in FIGS. 33A and 33B. While embodiments are primarilydescribed herein in relation to processing TCP packet flows, note thatembodiments may be applied to other data communications protocols thanTCP, and to other applications than processing packet flows.

The distributed load balancer may act to facilitate and maintain TCPpacket flows between particular clients and selected servers (e.g., webservers). However, the distributed load balancer does not terminate theTCP flows from the clients and does not act as a proxy to the servers asis done in conventional load balancers. Instead, the load balancer nodesof the distributed load balancer route TCP packets received from theclients to target servers, and the servers use their TCP stacks tomanage the TCP connections to the clients. In other words, the serversterminate the TCP packet flows from the clients.

In addition, instead of the load balancer node(s) making decisions as towhich server will service a connection request based on a load balancingtechnique or algorithm applied to information collected from the serversas is done in conventional load balancer technology, the load balancernodes may randomly select a server to receive a new connection request,and a component of the distributed load balancer that resides on theserver node makes the decision locally as to whether the selected serverwill accept or reject the new connection request based on one or moremetrics of the current status of the respective server. Thus, thedecisions as to which servers are to accept connection requests is movedfrom the load balancer node(s) to the server nodes that will be handlingthe connections. In other words, the decision is moved closer to whereand when the connection request will be serviced.

To facilitate and maintain the packet flows between the clients and theservers, embodiments of the distributed load balancer may employ varioustechniques or technologies including but not limited to multipathrouting technology, consistent hashing technology, distributed hashtable (DHT) technology, Border Gateway Protocol (BGP) technology,membership tracking, health checking, connection publishing, and packetencapsulation and decapsulation. These as well as other aspects of thedistributed load balancing system are described below in relation to theFigures.

Distributed Load Balancing System

FIG. 1 is a block diagram of an example distributed load balancingsystem, according to at least some embodiments. Embodiments of thedistributed load balancer may be implemented in a network 100, forexample a provider network 1900 of a service provider as illustrated inFIGS. 33A and 33B. As a high-level overview of client packet handling inthe distributed load balancer system, one or more clients 160 of thenetwork 100 may connect to a border router 102 of the network 100, forexample via an external network 150 such as the Internet. The borderrouter 102 may route incoming packets (e.g., TCP packets) from clients160 to an edge router 104 component of the distributed load balancerthat routes the incoming packets to the load balancer (LB) nodes 110 ina load balancer node layer of the distributed load balancer system. Inat least some embodiments, the edge router 104 may make the routingdecisions according to a per-flow hashed multipath routing technique,for example an equal-cost multipath (ECMP) hashing technique. The loadbalancer nodes 110 in turn encapsulate the packets (e.g., according toUser Datagram Protocol (UDP)) and route the encapsulated packets to thelocal load balancer modules 132 on the server nodes 130 via a networkfabric 120 (e.g., an L3 network) on network 100. The fabric 120 mayinclude one or more networking devices or components including but notlimited to switches, routers, and cables. On the server nodes 130, thelocal load balancer modules 132 decapsulate the packets and send theclient TCP packets to the TCP stacks of the servers 134. The servers 134on the server nodes 130 then use their TCP stacks to manage theconnections to the clients 160.

FIG. 2 is a high-level flowchart of a load balancing method that may beimplemented by the distributed load balancer system of FIG. 1, accordingto at least some embodiments. Embodiments of the distributed loadbalancer system may not solve the hard problem of assigning load amongmultiple destinations (e.g., web servers) as is done in conventionalload balancers. For example, conventional load balancers typically usetechniques or algorithms such as max connections, round robin, and/orleast connections techniques to select which server should handle aconnection. However, these techniques have drawbacks, and in particularare difficult to perform successfully in a distributed system where thedata used to make load balancing decisions is often almost immediatelystale. In at least some embodiments of the distributed load balancersystem, instead of attempting to select a server node 130 to satisfy aconnection request using one or more of the load balancing techniques asis done in conventional load balancers, a load balancer node 110 in theload balancer node layer may randomly determine a server node 130 toreceive a request for a client connection. If that server node 130considers itself overloaded, the server node 130 may send the connectionrequest back to the load balancer node 110 thus informing the loadbalancer node 110 that the server node 130 cannot currently handle theconnection. The load balancer node layer may then randomly determineanother server node 130 to receive the connection request, oralternatively may return an error message to the requesting client 160to inform the client 160 that the connection cannot currently beestablished.

As indicated at 10 of FIG. 2, the load balancer node layer of thedistributed load balancer system receives a request for a communicationsession (e.g., a TCP connection) from a source. The source may, forexample, be a client 160 on an external network 150 to the network 100that implements the distributed load balancer system. In at least someembodiments, the request may be received from the client 160 at a borderrouter 102 of the network 100, and routed to an edge router 104 thatroutes the incoming packets to the load balancer (LB) nodes 110 in aload balancer node layer, for example using a per-flow equal-costmultipath (ECMP) hashing technique to pseudorandomly select a loadbalancer node 110 to which a particular connection request from a client160 is to be routed.

As indicated at 20, the load balancer node layer randomly selects adestination node and forwards the connection request to the selecteddestination node. The destination node may, for example, be one of aplurality of server nodes 130 fronted by the load balancer. In at leastsome embodiments, a load balancer node 110 in the load balancer layermay randomly select a server node 130 to receive a connection requestfrom among all known server nodes 130. However, other methods thanpurely random selection from among all known server nodes 130 may beused in some embodiments to select server nodes 130 to receive theconnection requests. For example, in some embodiments, information aboutthe server nodes 130 may be used by the load balancer nodes 110 toweight the random selection of server nodes 130. As an example, if theload balancer nodes 110 know that different server nodes 130 aredifferent types of devices or are configured with different CPUs andthus have different capabilities or capacities, the information may beused to bias the random selection towards (or away from) particulartype(s) or configuration(s) of server node 130.

As indicated at 30, the destination node determines if it can accept thecommunications session. In at least some embodiments, a local loadbalancer (LB) module 132 on the server node 130 determines if therespective server 134 on the server node 130 can accept the newconnection based on one or more metrics of the current status of therespective server 134.

At 40, if the connection request is accepted, then as indicated at 50the destination node informs the load balancer node layer that thedestination node can handle the connection. As indicated at 60, acommunications session is then established between the source (e.g., aclient 160) and the destination node (e.g., a server 134 on a servernode 130) via the load balancer node layer. In at least someembodiments, the server 134 on the server node 130 uses a TCP stack tomanage the connection to the client 160.

At 40, if the connection request is not accepted, then as indicated at70 the destination node notifies the load balancer node layer, and themethod may return to element 20. The load balancer node layer may thenrandomly select another destination node at 20, or alternatively mayinform the requesting client 160 that the connection cannot currently beestablished. Note that the client 160 may, but does not necessarily,resubmit the connection request to begin the method again at element 10.

Referring again to FIG. 1, at least some embodiments of the distributedload balancer system may use commodity hardware to route client trafficreceived at an edge router 104 on network 100 to server nodes 130 on thenetwork 100. At least some embodiments of the distributed load balancermay include a load balancer node layer that includes multiple loadbalancer nodes 110. In at least some embodiments, each load balancernode 110 may serve in one or more of multiple roles in the load balancernode layer. These roles of the load balancer nodes 110 may include theroles of an ingress node, and egress node, and a flow tracker node (as aprimary flow tracker or a secondary flow tracker for a given packetflow). In at least some embodiments, each load balancer node 110 may beimplemented in the load balancer node layer as or on a separatecomputing device, such as a commodity rack-mounted computing device. Inat least some embodiments, each load balancer node 110 may serve in eachof the three roles of ingress node, egress node, and flow tracker node(as a primary or secondary flow tracker for a packet flow), with theload balancer node 110 generally serving in only one (but possibly intwo or three) of the roles for particular packet flows. Note, however,that in at least some embodiments, a load balancer node 110 is notallowed to serve as both the primary flow tracker and the secondary flowtracker for a particular packet flow. Alternatively, in someembodiments, each load balancer node 110 may serve in only one of thethree roles. In this embodiment, separate sets of computing devices maybe implemented in the load balancer node layer specifically as ingressnodes, egress nodes, and flow tracker nodes.

In at least some embodiments, consistent hashing and consistent hashring technology may be applied to determine the primary and secondaryflow trackers for the packet flows. Each packet flow from a client maybe uniquely identified, for example by a 4-tuple consisting of: theclient IP address, client port, server (public) IP address, and serverport. This identifier may be abbreviated as CP or CcPp indicating theclient and public endpoint pair. Packets associated with any given TCPflow (or CP pair) can appear on any load balancer node 110 operating asan ingress server 112 due to the hashed multipath (e.g., ECMP) flowdistribution from the edge router 104. Consistent hashing is used sothat when a packet arrives at a load balancer node 110 serving as aningress node, the ingress node can determine which load balancer node110 is responsible for maintaining the state for the packet flow (i.e.,the primary flow tracker node). The CP pair may be hashed by the ingressnode into a consistent hash ring to determine which load balancer node110 is responsible for maintaining state information for the packetflow. The node 110 determined according to the consistent hash of the CPpair for the packet flow in the consistent hash ring is the node 110that serves as the primary flow tracker for the packet flow. In at leastsome embodiments, the successor node in the consistent hash ring servesas the secondary flow tracker for the packet flow.

FIG. 3 shows an example load balancer (LB) node 110 that includescomponents that implement all three roles (ingress, egress, and flowtracker), according to at least some embodiments. In this example, aningress server 112 component performs the ingress role of receivinginbound TCP packets from client(s) and sending the TCP packets asencapsulated packets to the server(s). An egress server 114 componentperforms the egress role of receiving outbound encapsulated packets fromthe server(s) and sending the decapsulated TCP packets on to theclient(s). A flow tracker 116 component performs as a primary orsecondary flow tracker for one or more packet flows that are establishedbetween the client(s) 160 and the server(s) 134. The ingress server 112may also communicate with the flow tracker 116 on load balancer node110, or with the flow tracker 116 on another load balancer node 110 toinitiate a TCP connection between a client and one of the servers 134 inresponse to a connection request received from the respective client160, or to obtain mapping information for the packet flow. If theingress server does

Load Balancer Nodes

Referring again to FIG. 1, in at least some embodiments, the loadbalancer nodes 110 in the load balancer node layer receive clienttraffic (packets, e.g. TCP packets) from one or more routers 104 on thenetwork and encapsulate the packets according to a protocol (e.g., theUser Datagram Protocol (UDP)) used by the distributed load balancersystem on the fabric 120. The load balancer node layer then forwards theencapsulated packets to the destination server nodes 130 over fabric120. Each server node 130 includes a local module 132 that is acomponent of the load balancer system. The module 132 may be referred toherein as a load balancer module or simply LB module, and may beimplemented in software, hardware, or a combination thereof on theserver node 130. At each server node 130, the respective load balancermodule 132 decapsulates the packets and sends the TCP packets to a localTCP stack for normal TCP processing. In at least some embodiments, theload balancer node layer may maintain state information for everyclient-server TCP flow; however, the load balancer nodes 110 in the loadbalancer node layer may not interpret anything about the TCP flow. Eachflow is managed between the server 134 on the respective server node 130and the client 160. The distributed load balancer system insures thatthe TCP packets arrive at the correct destination server 134. The loadbalancer module 132 at each server node 130 makes the decision as towhether the respective server 134 will accept or reject a new connectionin response to a client connection request received from a load balancernode 110.

In at least some embodiments, the distributed load balancing system mayuse consistent hashing technology to, for example, determine which loadbalancer node(s) 110 should remember which server node 130 isresponsible for a particular TCP packet flow. Using consistent hashingtechnology, the load balancer nodes 110 in the load balancer node layermay be viewed as a consistent hash ring, and the load balancer nodes 110may keep track of membership in the ring and determine particularmembers in the ring that are responsible for particular packet flowsaccording to a consistent hashing function. In at least someembodiments, there are two load balancer nodes 110 that are responsiblefor tracking each packet flow between the clients 160 and the servers134; these nodes 110 may be referred to as the primary flow tracker(PFT) node and the secondary flow tracker (SFT) node. In at least someembodiments, the primary flow tracker is the first load balancer node110 on the consistent hash ring for the flow, and the secondary flowtracker is the next or subsequent load balancer node 110 on theconsistent hash ring distinct from the primary flow tracker node. Inthis arrangement, if the primary flow tracker node fails, then thesecondary flow tracker node may become the new primary flow tracker, andanother load balancer node 110 (e.g., the next node 110 on theconsistent hash ring) may assume the role of the secondary flow tracker.Note that, in at least some embodiments, a load balancer node 110 is notallowed to serve as both the primary flow tracker and the secondary flowtracker for a given packet flow. This and other membership changes inthe consistent hash ring are discussed later in this document. In atleast some embodiments, configuration information for the load balancerimplementation (e.g., authoritative list(s) of the load balancer nodes110 and server nodes 130 that are currently in the implementation) maybe maintained by a configuration service 122 component of thedistributed load balancing system, which may for example be implementedon one or more server devices coupled to the load balancer nodes 110 viathe fabric 120.

In at least some embodiments, in addition to serving as primary andsecondary flow tracker nodes, the load balancer nodes 110 may alsoperform in one of two other roles for a given flow: the role of aningress node and the role of an egress node. An ingress node for apacket flow is the load balancer node 110 that receives the respectivepacket flow from the edge router 104 and forwards the packet flow (asencapsulated packets) to a selected server 134 on a server node 130 viafabric 120. An ingress node is the only load balancer node 110 thatmoves actual client data (TCP data packets) to the respectivedestination server node 130. The ingress node maintains a mapping of theTCP flow to a respective load balancer module 132 on the destinationserver node 130 so that the ingress node knows which load balancermodule 132 to forward the client traffic to. An egress node is a loadbalancer node 110 that is responsible for forwarding the responsetraffic for a packet flow received from the server node 130 via fabric120 to the respective client 160 via the border network. The loadbalancer module 132 encapsulates response packets obtained from theserver 134 according to a load balancer protocol (e.g., UDP) and sendsthe encapsulated response packets to the respective egress node for theflow via fabric 120. The egress nodes are stateless and simplydecapsulate the packets and send the response packets (e.g., TCPpackets) onto the border network to a border router 102 for delivery tothe respective client 160 via the external network 150.

As previously mentioned, in at least some embodiments, each loadbalancer node 110 may perform the roles of an ingress node, an egressnode, and/or a flow tracker node (as either a primary or a secondaryflow tracker) for different packet flows. A single load balancer node110 in the load balancer node layer may perform in any one of the rolesdepending on what packet flow the node is processing. For example, in atleast some embodiments, a load balancer node 110 may perform as aningress node for one packet flow, as a primary or secondary flow trackerfor another packet flow, and as an egress node for yet another packetflow. In addition, in at least some embodiments a load balancer node 110may perform multiple roles for the same packet flow, for example as theingress node and as the primary (or secondary) flow tracker node for agiven packet flow. However, in at least some embodiments, for redundancyand recovery purposes, a load balancer node 110 is not allowed to serveas both the primary and secondary flow tracker node for the same packetflow.

The above describes embodiments where each load balancer node 110 mayserve in any of the three roles of ingress server, egress server, andflow tracker. However, in some embodiments, different groups ofcomputing devices may be assigned to the different roles in the loadbalancing system. For example, in some embodiments, there may bedistinct sets of ingress nodes, egress nodes and flow tracker nodes eachimplemented on a separate computing device. As another example, in someembodiments, one set of computing devices may serve as both ingressnodes and flow tracker nodes, while another set of computing devices mayserve only as egress nodes.

Load Balancer Modules

As previously mentioned, each server node 130 includes a local loadbalancer module 132 that is a component of the load balancer system. Themodule 132 may be implemented in software, hardware, or a combinationthereof on the server node 130. In at least some embodiments, the loadbalancer module 132 on a server node 130 may perform three main roles:encapsulating outgoing and decapsulating incoming packets, making localload balancing decisions for the server 134 on the node 130, andconnection publishing. These three roles are briefly described below,and described in more detail later in this document.

At least some embodiments of the distributed load balancing system donot terminate TCP connections and do not spoof packets; the source anddestination IP addresses of all packets sent through the load balancernode layer are the actual IP addresses of the endpoints (i.e., theclients 160 and servers 134) involved in the packet flows. Instead ofspoofing, these embodiments encapsulate all packets sent between theload balancer nodes 110 and the server nodes 130 on fabric 120, forexample as UDP packets. Since the inbound packets in a packet flowarriving at a server node 130 from a load balancer node 110 acting asthe ingress node for the flow are encapsulated by the load balancer node110, the packets need to be decapsulated and redirected to a localhostTCP flow for the server 134 on the node 130. The load balancer module132 on node 130 performs this decapsulation. Similarly, outgoing packetsfor a packet flow from the server 134 are encapsulated by the loadbalancer module 132 and sent via fabric 120 to a load balancer node 110acting as the egress node for the packet flow.

In at least some embodiments, the load balancer modules 132 on theserver nodes 130 also make local decisions related to load balancing forthe servers 134 on the respective server nodes 130. Specifically, theload balancer module 132 on a node 130 decides whether the respectiveserver 134 will accept another TCP flow in response to receiving arequest for a new TCP connection. As previously noted, the load balancernodes 110 encapsulate all packets sent to the load balancer module 132,so the load balancer module 132 actually does not receive a TCPsynchronize (SYN) packet from the client 160; instead, the load balancermodule 132 receives a connection request message according to theencapsulation protocol (e.g., UDP) from a flow tracker 116 which theload balancer module 132 can either accept or reject. If the loadbalancer module 132 accepts the connection request message, the loadbalancer module 132 creates a SYN packet destined for the localhost.When the localhost accepts the connection, this becomes the actual TCPstack handling the respective client connection.

In at least some embodiments, to make the decision as to whether aconnection request message should be accepted, the load balancer module132 looks at one or more metrics regarding current resource consumptionon the server node 130, and if there are sufficient resources availableto handle the new connection, the load balancer module 132 accepts theconnection. In at least some embodiments, resource metrics that may beconsidered by the load balancer module 132 may include one or more of,but are not limited to, CPU utilization, recent bandwidth consumption,and number of established connections. Other metrics may be consideredinstead of or in addition to these metrics in some embodiments. Forexample, in some embodiments, the load balancer module may considerserver latency (i.e., the amount of time requests are spending in theserver connection backlog) as a metric, and may reject the connectionrequest if server latency is above a threshold. Using these and/or othermetrics, the load balancer module 132 can decide for the respectiveserver 134 whether the server 134 is to accept or reject new packetflows. In at least some embodiments, a resource utilization rate (e.g.,N % utilization) may be determined from the metric(s) individually or incombination and compared to a threshold (e.g., 90% utilization). If thedetermined resource utilization rate is at or above the threshold, or ifadding the connection would move the rate to above the threshold, thenthe connection request may be rejected.

In at least some embodiments, the load balancer modules 132 mayimplement a probabilistic method for determining if connection requestmessages are to be rejected. Instead of rejecting all connectionrequests if resource utilization is at or above a threshold as describedabove, in this method may reject connection requests at differentprobabilities at two or more different levels of utilization. Forexample, if resource utilization is 80%, a load balancer module 132 mayreject connection requests at 20% probability; if resource utilizationis 90%, the load balancer module 132 may reject connection requests at25% probability; if resource utilization is 95%, the load balancermodule 132 may reject connection requests at 50% probability; and at 98%or above, the load balancer module 132 may reject all connectionrequests.

In at least some embodiments, each connection request message mayinclude an indication of how many times the connection request messagehas been rejected by load balancer modules 132. If a connection requestmessage received by a load balancer module 130 indicates that it hasbeen rejected over a threshold number of times, the load balancer module130 may accept the connection even though the performance metrics of theserver node 130 indicate that the connection request should be rejected.

In some cases, it is possible that all of the load balancer modules 132that a connection request message is sent to may reject the connectionrequest. In at least some embodiments, to prevent a connection requestmessage from being bounced from load balancer module 132 to loadbalancer module 132 for an indefinite period, each connection requestmessage may be given a time to live. If this time to live expires, theflow tracker node may terminate the request and notify the respectiveclient 160 that the request cannot currently be serviced.

In at least some embodiments, the load balancer modules 132 on theserver nodes 130 also perform connection publishing to the load balancernodes 110. In at least some embodiments, to perform connectionpublishing, periodically or aperiodically (e.g., once a second) eachload balancer module 132 looks at the routing table (e.g., a netstatrouting table) on the server node 130 and publishes a list of activeconnections (TCP flows) back to the load balancer nodes 110. The loadbalancer nodes 110 that need to be informed about the existence of agiven packet flow are the load balancer nodes 110 that are serving asthe ingress node and as the primary and secondary flow trackers for therespective packet flow. In some embodiments, the load balancer module132 may use a consistent hashing technique to filter the list of loadbalancer nodes 110 that need to be informed about the active TCP flowson the server node 130. For example, the load balancer module 132 maydetermine which load balancer nodes 110 are serving as the primary andsecondary flow trackers for a given packet flow according to theconsistent hash ring. In some embodiments, the load balancer module 132tracks which load balancer node 110 last sent a data packet to the loadbalancer module 132 for each packet flow, and uses this information todetermine which load balancer nodes 110 are serving as ingress nodes forthe packet flows, since only ingress nodes forward client data to theload balancer module 132. In some embodiments, the load balancer module132 then formulates messages for each of the load balancer nodes 110that it has determined need to be informed about the packet flows andsends the messages to the load balancer nodes 110 to inform the nodes110 that the respective server node 130 is still maintaining theconnection(s) to the client(s) 160. This connection publishing to theload balancer nodes 110 by the load balancer modules 132 may be viewedas extending a lease at the load balancer nodes 110. If a load balancernode 110 has not received a connection publishing message indicating aparticular packet flow within a period of time (e.g., ten seconds), thenthe load balancer node 110 is free to forget about the respective packetflow.

Multipath Routing to Load Balancer Nodes

FIG. 4 illustrates aspects of routing and packet flow in the distributedload balancer, according to at least some embodiments. In at least someembodiments, each ingress node (ingress nodes are shown in FIG. 4 asingress servers 112) advertises its ability to route one or more publicendpoints (e.g., IP address and port) to the edge router 104 for thedistributed load balancer, for example via the border gateway protocol(BGP). In at least some embodiments, rather than each ingress nodeadvertising itself to the edge router 104 via a BGP session, one or moreother ingress nodes, for example two neighbor nodes, may establish BGPsessions with the edge router 104 to advertise the ingress node, asshown in FIG. 5.

Conventional load balancers can typically only serve a single publicendpoint. In contrast, embodiments of the distributed load balancerenable multiple load balancer nodes 110 to service a single publicendpoint. Depending on router capabilities, this enables configurationsin which a single public IP address routed to all the ingress servers112 may handle the entire bandwidth (e.g., 160 Gbps) through the edgerouter(s) 104. In at least some embodiments, to achieve this, the edgerouter(s) 104 may utilize a layer 4 per-flow hashed multipath routingtechnique, for example an equal-cost multipath (ECMP) routing technique,to distribute traffic across multiple ingress servers 112 eachadvertising the same public IP address. Distributing incoming packets toall of the ingress servers 112 using layer-4 source and destinationports for the flows as part of the edge router(s) 104 flow hash maygenerally keep the packets for each connection routed to the same loadbalancer node 110 serving as the ingress server 112 to avoidout-of-order packets. Note, however, that the edge router(s) 104 may useother techniques to distribute traffic across the ingress servers 112 insome embodiments.

FIG. 4 also shows that two or more distributed load balancers may beimplemented on a network 100. The two or more distributed load balancersmay each act as an independent load balancer that fronts a plurality ofservers 130 and that each advertises a different public IP address, oralternatively as shown in FIG. 4 two or more distributed load balancersmay each advertise the same IP address, and a hashing technique (e.g., alayer 4 per-flow hashed multipath routing technique) may be used at theborder router(s) 102 to partition the packet flows out to the edgerouters 104, which in turn distribute the packet flows to theirrespective ingress servers 112.

FIG. 5 illustrates using Border Gateway Protocol (BGP) to advertiseingress nodes to the edge router, according to at least someembodiments. In this example, there are four load balancer nodes servingas ingress nodes 110A through 110D in the load balancer implementation.Edge router 104 routes incoming packets from clients (not shown) to theload balancer nodes 110. In at least some embodiments, the edge router104 may make the routing decisions according to a layer 4 per-flowhashed multipath routing technique, for example an equal-cost multipath(ECMP) routing technique.

In at least some embodiments, edge router 104 learns about the ingressnodes 110 that are currently available in the load balancerimplementation to receive client traffic via Border Gateway Protocol(BGP) technology advertising sessions initiated by the ingress nodes110. Each ingress node 110 could use BGP to advertise itself to the edgerouter 104. However, BGP typically takes a relatively long time toconverge (three seconds or more). Using this technique where eachingress node 110 advertises itself via BGP, if an ingress node 110 goesdown, it may take considerable time in networking terms (three secondsor more) for the BGP session on the edge router 104 to time out and thusfor the edge router 104 to learn about the failure close down andreroute the current TCP flows to the ingress node 110.

To avoid the convergence problem with BGP and to recover more quicklyupon node 110 failure, in at least some embodiments, instead of aningress node 110 advertising itself to the edge router 104 via a BGPsession, at least one other ingress node 110 in the load balancerimplementation takes responsibility for advertising the ingress node 110to the edge router 104 via BGP. For example, in some embodiments asshown in FIG. 5, the left and right neighbor ingress nodes 110 of agiven ingress node 110, for example the left and right neighbors in anordered listing of the nodes 110, for example a consistent hash ringformed by the nodes 110, may advertise the given ingress node 110 to theedge router 104. For example, in FIG. 5, ingress node 110A advertisesingress nodes 110B and 110D, ingress node 110B advertises ingress nodes110A and 110C, ingress node 110C advertises ingress nodes 110B and 110D,and ingress node 110D advertises ingress nodes 110C and 110A. Theingress nodes 110 check and gossip each other's health as describedlater in this document. Using the health check method as described,unhealthy nodes can be detected and the information can be propagatedamong the nodes 110 in less than a second, for example in 100milliseconds (ms). Upon determining that an ingress node 110 is nothealthy, the ingress nodes 110 that advertise the unhealthy node mayimmediately stop advertising the unhealthy node 110. In at least someembodiments, the ingress nodes 110 end the BGP sessions with the edgerouter 104 by sending a TCP Close or similar message for the BGP sessionto the edge router 104. Thus, rather than having to wait for a BGPsession established by a failed node 110 to time out to detect the node110 failure, the edge router 104 may discover the failed node 110 whenthe other ingress nodes 110 that advertise on behalf of the failed node110 terminate the BGP sessions with the edge router 104 that advertisethe node 110 upon detecting that the node 110 is unhealthy. The handlingof load balancer node failures is further discussed in relation to FIGS.18A and 18B later in this document.

FIG. 6 is a flowchart of a multipath routing method, according to atleast some embodiments of the distributed load balancing system. Asindicated at 900, the ingress nodes 110 in a load balancerimplementation advertise their neighbor nodes 110 to the edge router104. In at least some embodiments, the ingress nodes 110 may determinetheir neighbor nodes 110 according to an ordered listing of the nodes110 such as a consistent hash ring. In at least some embodiments, theingress nodes 110 advertise their neighbor node(s) 110 to the edgerouter 104 using BGP sessions, with one BGP session established to theedge router 104 for each advertised node 110.

As indicated at 902, the edge router 104 distributes traffic receivedfrom clients 160 to the active (advertised) ingress nodes 110 accordingto a per-flow hashed multipath routing technique, for example anequal-cost multipath (ECMP) routing technique. In at least someembodiments, the edge router 104 exposes a public IP address to theclients 160; the ingress nodes 110 all advertise the same public IPaddress to the edge router 104. The edge router uses layer-4 source anddestination ports as part of the edge router's 104 flow hash todistribute the incoming packets among the ingress nodes 110. Thisgenerally keeps the packets for each connection routed to the sameingress node 110.

As indicated at 902, the ingress nodes forwards the data flows to targetserver nodes 130. In at least some embodiments, the ingress nodes 110interact with primary and secondary flow tracker nodes for the dataflows to map the data flows to the target server nodes 130. Thus, eachingress node 110 may maintain mappings of active data flows through thenode 110 that may be used to appropriately forward the received packetsto the target server nodes 130.

Elements 906 through 910 relate to detecting and recovering from ingressnode 110 failures. As indicated at 906, the ingress nodes 110 may detectthat an ingress node 110 is down, for example according to a healthchecking technique as described herein. Upon detecting that the node 110is down, its neighbor nodes 110 stop advertising the node 110 to theedge router 104. In at least some embodiments, this involves sending aTCP Close to the edge router 104 for the respective BGP session.

As indicated at 908, the edge router 104, upon detecting that theingress node 110 is down via the closing of the BGP sessions,redistributes incoming traffic from the clients 160 to the remainingingress nodes 110 according to the per-flow hashed multipath routingtechnique. Thus, at least some data flows may be routed to differentingress nodes 110.

As indicated at 910, the ingress nodes 110 may recover mappings asnecessary and forward the data flows to the appropriate target servernodes. Methods for recovering from node 110 failures on ingress nodes110 are discussed elsewhere in this document. As one example, an ingressnode 110, upon receiving a packet for which it does not have a currentmapping, may use a consistent hash function to determine a flow trackernode for the data flow according to a consistent hash ring and recoverthe mapping from the flow tracker node.

Asymmetric Packet Flow

In at least some embodiments, to efficiently utilize ingress nodebandwidth and CPU usage when the ratio of outbound traffic to inbounddata is greater than 1, the distributed load balancing system forwardsoutbound packets from the server nodes 130 to multiple egress nodes asshown in FIG. 7. In at least some embodiments, for each connection, theload balancer module 132 on the respective server node 130 hashes theclient endpoint/public endpoint tuple and uses a consistent hashalgorithm to select a load balancer node 110 to serve as the egressserver 114 for the respective outbound packet flow. However, in someembodiments other methods and/or data may be used to select the egressservers 114 for connections. The selected egress server 114 maytypically be, but is not necessarily, a different load balancer node 110than the load balancer node 110 that serves as the ingress server 112for the connection. In at least some embodiments, unless there is afailure of that load balancer node 110/egress server 114, all of theoutbound packets for the particular connection will be forwarded to thesame egress server 114 in order to avoid out-of-order packets.

In at least some embodiments, the method and data used for selecting anegress server 114 by the server nodes 130 may be different than themethod and data used for selecting an ingress server 112 performed bythe edge router(s) 104. Using the different methods and data maygenerally result in a different load balancer node 110 being selected asthe egress node for a given connections than the load balancer node 110selected as the ingress node for the connection, and may also result inmultiple load balancer nodes 110 being selected as egress nodes tohandle the outgoing traffic for connections that pass through a singleload balancer node 110 serving as an ingress node.

FIG. 7 graphically illustrates asymmetric packet flow, according to atleast some embodiments. At least one connection has been establishedfrom clients 160 on external network 150 through ingress server 112 toeach of server nodes 130A, 130B, 130C, and 130D. In at least someembodiments, to select egress nodes for the connections, for eachconnection, the load balancer module 132 on the respective server node130 hashes the client endpoint/public endpoint tuple and uses aconsistent hash algorithm to select a load balancer node 110 to serve asthe egress server 114 for the respective outbound packet flow. Forexample, server node 130A has selected egress server 114A for aconnection, and server node 114B has selected egress server 114A for oneconnection and egress server 114B for another connection. However, insome embodiments other methods and/or data may be used to select theegress nodes for connections.

Recovering from Load Balancer Node Failures without Dropping ClientConnections

While it is possible for the load balancer nodes 110 to use consistenthashing to determine which server node 130 should receive clienttraffic, due to the long lifespan of some connections this approach maynot maintain existing flows in cases where a new server node 130 joinsthe consistent hash membership and there is a subsequent ingress loadbalancer node 110 failure. In this scenario, a load balancer node 110that takes over a flow from the failed node 110 may not be able todetermine the original mapping selected, as the consistent hash ring forthe servers 130 would have different membership. Thus, in at least someembodiments, distributed hash table (DHT) technology may be used by theload balancer nodes 110 to select server nodes 130 for connections andto route packets to the selected server nodes 130. Once a server node130 has been selected according to the DHT to receive a particularconnection, and assuming that the server node 130 stays healthy and thatthe load balancer module 132 on the server node 130 continues to extendthe lease by periodically transmitting the status of that activeconnection to the DHT (e.g., via connection publishing), the DHT willretain the mapping until the connection completes. An ingress node 110failure impacts the distribution of packets from the edge router 104 tothe remaining load balancer nodes 110, resulting in the load balancernodes 110 receiving traffic from a different set of client connections.However, since the DHT tracks all active connections, the load balancernodes 110 can query the DHT to obtain leases for any active mappings. Asa result, all the load balancer nodes 110 will pass traffic to thecorrect server nodes 130, thus preventing failure of active clientconnections even in the event of an ingress load balancer node 110failure.

Packet Flow in the Distributed Load Balancing System

FIG. 8 illustrates packet flow in the distributed load balancing system,according to at least some embodiments. Note that the solid lines witharrows in FIG. 8 represent TCP packets, while the dotted lines witharrows represent UDP packets. In FIG. 8, an ingress server 112 receivesTCP packets from one or more clients 160 via the edge router 104. Uponreceipt of a TCP packet, ingress server 112 determines if it has amapping for the TCP packet flow to a server node 130. If the ingressserver 112 does have a mapping for the TCP packet flow, then the server112 encapsulates the TCP packet (for example according to UDP) and sendsthe encapsulated packet to the target server node 130. If the ingressserver 112 does not have a mapping for the TCP packet flow, then theingress server 112 may send a UDP message including information aboutthe TCP packet flow extracted from the TCP packet to the primary flowtracker 116A to establish a connection to a server node 130 and/orobtain a mapping for the TCP packet flow. FIGS. 9A and 9B and FIGS. 10Athrough 10G illustrate methods for establishing a connection between aclient 160 and a server node 130. The load balancer module 132 on aserver node 130 randomly selects load balancer node(s) 110 to serve asthe egress server(s) 114 for TCP connection(s) on the server node 130and sends UDP-encapsulated TCP response packets to the client(s) 160 viathe egress server(s) 114.

FIGS. 9A and 9B provide a flowchart of packet flow when establishingconnections in the distributed load balancing system, according to atleast some embodiments. As indicated at 200 of FIG. 9A, an ingressserver 112 receives a TCP packet from a client 160 via the edge router104. At 202, if the ingress server 112 has a mapping for the TCP flow toa server node 130, then the ingress server 112 encapsulates and sendsthe TCP packet to the respective server node 130 as indicated at 204.Note that the ingress server 112 may be continuously receiving andprocessing packets for one, two or more TCP flows from one, two, or moreclients 160.

At 202, if the ingress server 112 does not have a mapping for the TCPflow, the packet may be a TCP synchronize (SYN) packet from a client160. As indicated at 206, upon receipt of a SYN packet, the ingressserver 112 extracts data from the SYN packet and forwards the data tothe primary flow tracker 116A, for example in a UDP message. In at leastsome embodiments, the ingress server 112 can determine the primary flowtracker 116A and/or secondary flow tracker 116B for the TCP flowaccording to a consistent hash function. At 208, the primary flowtracker 116A stores the data, for example in a hash table, generates aninitial TCP sequence number for the server node 130 side of the TCPconnection, and forwards the data and the TCP sequence number to thesecondary flow tracker 116B. At 210, the secondary flow tracker 116B mayalso store the data, and fabricates and sends a SYN/ACK packet to theclient 160, the SYN/ACK packet containing at least the TCP sequencenumber.

As indicated at 212, the ingress server 112 receives a TCPacknowledgement (ACK) packet from the client 160 via the edge router104. The ingress server 112 does not at this time have a mapping for theTCP flow to a server 130 node, so at 214 the ingress server 112 sends amessage including data extracted from the ACK packet to the primary flowtracker 116A. As indicated at 216, upon receiving the message, theprimary flow tracker 116A confirms the TCP flow according to the storeddata, and confirms that the acknowledged sequence number (+1) from theACK packet matches the value sent in the SYN/ACK. The primary flowtracker 116A then selects a server node 130 to receive the TCP flow, andsends a message containing the data, TCP sequence number, and IP addressof the local load balancer module 132 on the selected server node 130 tothe secondary flow tracker 116B. As indicated at 218, the secondary flowtracker 116B also confirms the data and TCP sequence number, fabricatesa SYN message, and sends the fabricated SYN message to the local loadbalancer module 132 on the selected server node 130. The methodcontinues at element 220 of FIG. 9B.

As indicated at 220 of FIG. 9B, in response to the fabricated SYNmessage, the load balancer module 132 may examine one or more metrics ofthe server node 130 to determine if the server node 130 can accept theconnection. At 222, if the load balancer module 132 determines that theserver node 130 cannot currently accept the connection, then at 224 theload balancer module 132 messages the secondary flow tracker 116B. Thesecondary flow tracker 116B may delete the information for the flow thatit previously stored. At 226, the secondary flow tracker 116B messagesthe primary flow tracker 116A. The primary flow tracker 116A may thenselect a new target server node 130 and message the secondary flowtracker 116B as indicated at 216 of FIG. 9A.

At 222, if the load balancer module 132 determines that the server node130 can accept the connection, then as indicated at 228 of FIG. 9B thelocal load balancer module 132 constructs a TCP SYN packet from thefabricated SYN and sends the TCP SYN packet to the server 134 on theserver node 130. The source IP address of the TCP SYN packet ispopulated with client 160's actual IP address so that server 134 willbelieve that it has received a direct TCP connection to the client 160.The load balancer module 132 stores relevant details about the TCP flow,for example in a local hash table. As indicated at 230, the server 134responds with a SYN/ACK packet that the load balancer module 132intercepts. As indicated at 232, the load balancer module 132 then sendsa message including connection information to the secondary flow tracker116B to indicate that the connection has been accepted. Upon receipt ofthis message, at 234 the secondary flow tracker 116B records the mappingto the server 134, and sends a similar message to the primary flowtracker 116A, which also records the mapping information. As indicatedat 236, the primary flow tracker 116A then forwards a mapping message tothe ingress server 112. Ingress server 112 now has a mapping for the TCPflow from the client 160 to the server 130.

At 238, ingress server 112 encapsulates and forwards any buffered datapackets for the data flow to the local load balancer module 132 on theserver node 130. Additional incoming packets for the data flow from theclient 160 received by the ingress server 112 are encapsulated andforwarded directly to the load balancer module 132, which decapsulatesthe packets and sends the data packets on to the server 134.

At 240, the load balancer module 132 randomly selects an egress server114 for the data flow. Subsequent outbound TCP packets from the server134 are intercepted by the load balancer module 132, encapsulatedaccording to UDP, and forwarded to the arbitrarily selected egressserver 114. The egress server 114 decapsulates the outgoing packets andsends the TCP packets to the client 160.

As noted above, at 202, if the ingress server 112 does not have amapping for the TCP flow of a received packet, the packet may be a TCPsynchronize (SYN) packet from a client 160. However, the packet may notbe TCP SYN packet. For example, if load balancer node 110 membershipchanges due to addition or failure of a load balancer node 110, the edgerouter 104 may start routing packets for one or more TCP flows to theingress server 112 that the ingress server 112 does not have mappingsfor. In at least some embodiments, upon receiving such a packet forwhich the ingress server 112 does not have a mapping, the ingress server112 may use the consistent hash function to determine the primary flowtracker 116A and/or secondary flow tracker 116B for the TCP flowaccording to the consistent hash ring and message either the primaryflow tracker 116A or the secondary flow tracker 116B to request themapping. Upon receiving the mapping for the TCP flow from a flow tracker116, the ingress server 112 can store the mapping and beginencapsulating and forwarding the TCP packet(s) for the TCP flow to thecorrect destination server node 130.

Load Balancer Node Details

In at least some embodiments, the load balancer nodes 110 each havethree roles:

-   -   Ingress—Receiving all incoming packets from a client 160 in a        client connection, routing the packets to a server node 130 if        the mapping is known, or messaging a flow tracker if the mapping        is not known. The outgoing packets from an ingress node are        encapsulated (e.g., according to UDP) by the ingress node.    -   Flow tracking—Keeping track of connection states (e.g. which        server node 130/server 134 has been assigned to service each        client connection). Flow trackers also participate in        establishing connections between clients 160 and servers 134.    -   Egress—Decapsulating and forwarding outbound packets received        from a server 134 to a client 160.

In at least some embodiments, in the ingress role, a load balancer node110 is responsible for forwarding packets to servers 134 when aclient->server mapping is known, or forwarding a request to a flowtracker when the mapping is unknown. In at least some embodiments, aload balancer node 110 serving as an ingress node for a particularclient connection/data flow may also serve as either the primary flowtracker or the secondary flow tracker for the client connection, but notboth.

In at least some embodiments, in the flow tracker role, a load balancernode 110 is responsible for maintaining the state of connections thatare still being established, as well as maintaining the client->servermapping for established connections. Two flow trackers are involved witheach individual client connection, referred to as the primary flowtracker and the secondary flow tracker. In at least some embodiments,the flow trackers associated with client connections may be determinedusing a consistent hash algorithm. The flow trackers also performload-balancing functionality, including but not limited topseudorandomly selecting a server node 130 for each new clientconnection. Note that the local load balancer module 132 on a selectedserver node 130 may reject a connection request if it determines thatthe server 134 cannot handle the connection. If this happens, then theflow trackers may select another server node 130 and send the connectionrequest to the other server node 130. In at least some embodiments, theprimary flow tracker role and the secondary flow tracker role for agiven connection are performed by different load balancer nodes 110.

In at least some embodiments, in the egress role, a load balancer node110 is stateless and decapsulates incoming packets received from servernodes 130, performs some validation, and forwards the outbound TCPpackets to respective clients 160. In at least some embodiments, a localload balancer module 132 on a server node 130 may arbitrarily select aload balancer node 110 for a given connection.

Load Balancer Node Consistent Hash Ring Topology

In at least some embodiments, the load balancer nodes 110 form a ringtopology based on the consistent hashing of input keyspace (clientendpoint, public endpoint). The input keyspace may be partitioned amongthe available flow tracker nodes, and every flow tracker node may beresponsible for answering queries corresponding to its keyspace. In atleast some embodiments, data may be replicated to the primary andsecondary flow tracker nodes based on the successor in the consistenthash ring (e.g., the secondary flow tracker node is the successor node,or next node in the consistent hash ring, to the primary flow trackernode). If a flow tracker node goes down for some reason, the next loadbalancer node in the consistent hash ring acquires the keyspace of thefailed node. When a new flow tracker node joins, the node registers itsendpoint (e.g., with a configuration service 122 as shown in FIG. 1) sothat other load balancer nodes may learn about the configuration changein the load balancer implementation and thus in the consistent hashring. The handling of additions and failures of flow trackers in theconsistent hash ring is discussed in more detail in reference to FIGS.11A through 11D.

Ingress Node<->Flow Tracker Node Communications

In at least some embodiments, the load balancer nodes 110 serving asingress nodes may learn about the load balancer nodes 110 serving asflow tracker nodes from configuration service 122. The ingress nodes maymonitor the configuration service 122 for membership changes in the loadbalancer implementation and thus in the consistent hash ring. When aningress node receives a packet from a client 160 that the ingress nodedoes not have a mapping for, the ingress node may use a consistent hashfunction to determine which flow tracker node should service the packet.In at least some embodiments, the input to the hash function is the(client endpoint, public endpoint) pair from the packet. In at leastsome embodiments, the ingress nodes and flow tracker nodes communicateusing UDP messages.

When a primary flow tracker node receives a message from an ingress nodefor a new packet flow, the primary flow tracker node randomly determinesa TCP sequence number and forwards another message to the secondary flowtracker node. The secondary flow tracker node generates a TCP SYN/ACKmessage for the client. Both flow trackers remember the clientconnection endpoint pair and the TCP sequence number, and retain thisinformation until memory pressure or expiration causes the state to bepurged.

When the primary flow tracker node receives a message from an Ingressnode that a TCP ACK packet has been received, the primary flow trackernode verifies that the acknowledged TCP sequence number matches thestored value that was sent in the SYN/ACK packet, selects a server node130 to service the request, and forwards a message to the secondary flowtracker node. The secondary flow tracker node sends a message to theload balancer module 132 on the selected server node 130 to initiate anactual TCP connection with the TCP stack on the server node 130, andthen waits for an acknowledgement response from the server node 130.

When the secondary flow tracker node receives a connectionacknowledgement from the load balancer module 132 on the server node130, a reverse message flow through the primary flow tracker to theingress node is triggered that stores information about the associatedserver node 130 in both nodes. From this point forward, additional TCPpackets received at the ingress node are forwarded directly to the loadbalancer module 132 on the server node 130.

Load Balancer Module<->Load Balancer Node Communications

In at least some embodiments, every load balancer module 132 registersits endpoint with configuration service 122 and monitors configurationservice 122 continuously for membership changes in the load balancernode layer. The following describes functions of the load balancermodule 132, according to at least some embodiments:

-   -   Connection publishing—periodically (e.g., once a second) or        aperiodically publish the set of active connections (client        endpoint, public endpoint) on the respective server node 130 to        both the primary and the secondary flow tracker nodes        responsible for those connections, as well as to the ingress        nodes that last sent packets to the load balancer module 132 for        those connections. The connection publishing function renews the        lease for the connection states at the responsible load balancer        nodes 110.    -   Monitor membership changes in the load balancer layer. If the        membership changes, the load balancer modules 132 may use this        change information to immediately send active connections to the        load balancer nodes that are now responsible for the        connections.

Packet Flow in the Distributed Load Balancing System—Details

The distributed load balancing system may include multiple load balancernodes 110. In at least some embodiments, each load balancer node 110 inthe distributed load balancing system may serve in the roles of a flowtracker node, an egress node, and an ingress node for client 160connections to the servers 134. The distributed load balancing systemmay also include a load balancer module 132 on each server node 130.

FIGS. 10A through 10G illustrate packet flow in the distributed loadbalancing system, according to at least some embodiments. In FIGS. 10Athrough 10G, packets exchanged between load balancer nodes 110 andpackets exchanged between load balancer nodes 110 and server nodes 130are either UDP messages or UDP-encapsulated client TCP packets. In atleast some embodiments, client TCP packets only exist on network 100 indecapsulated form on the north side of the load balancer nodes 110 intransit to and from the border router 102 (see FIG. 1). Note that thesolid lines with arrows in FIGS. 10A-10G represent TCP packets, whilethe dotted lines with arrows represent UDP packets.

In at least some embodiments, the distributed load balancing system mayattempt to preserve established connections in the event of a singleload balancer node 110 failure. In at least some embodiments, this maybe achieved by replicating connection details in a primary flow trackernode and a secondary flow tracker node so that, if either of these nodesfails, a connection's client->server mapping may be restored by theremaining flow tracker node. In at least some embodiments, some packetloss may occur in the event of a node failure; however, client/serverTCP packet retransmissions may recover the lost packets.

Each TCP connection from a client may be referred to as a TCP flow, andis uniquely identified by a 4-tuple consisting of: the client IPaddress, client port, server (public) IP address, and server port. Thisidentifier may be abbreviated as CP or CcPp indicating the client andpublic endpoint pair. Packets associated with any given TCP flow (or CPpair) can appear on any load balancer node 110 operating as an ingressserver 112 due to the hashed equal-cost multipath (ECMP) flowdistribution from the upstream edge router 104. However, packets for aTCP flow may generally continue to arrive at the same load balancer node110 unless there is a link or load balancer node 110 failure that causesthe TCP flows to be redirected. The load balancer node 110 that receivespackets for a TCP flow from the upstream router 104 is referred to asthe ingress node for the TCP flow.

In at least some embodiments, consistent hashing is used so that whenpackets arrive at a load balancer node 110 serving as an ingress nodefor the TCP flow, the ingress node can determine which load balancernode 110 contains the state for the TCP flow (i.e., the flow trackernode). The CP pair may be hashed by the ingress node into a consistenthash ring to determine which load balancer node 110 is responsible formaintaining state regarding the TCP flow. This node serves as theprimary flow tracker for the TCP flow. The successor node in theconsistent hash ring serves as the secondary flow tracker for the TCPflow.

In at least some embodiments, all load balancer nodes 110 may serve asingress nodes, primary flow tracker nodes, and secondary flow trackernodes. Depending on the consistent hash result for a TCP flow, a loadbalancer node 110 serving as the ingress node for the TCP flow may alsoserve as the primary or the secondary flow tracker node for the TCPflow. However, in at least some embodiments, different physical loadbalancer nodes 110 perform the primary and secondary flow tracker rolesfor the TCP flow.

Establishing Connections

Referring to FIG. 10A, new connections from a client 160 may betriggered by a client TCP synchronize (SYN) packet. The load balancernodes 110 do not actually establish a connection with a server node 130upon receipt of the SYN packet, nor do they immediately select a servernode 130 to receive the connection. Instead, the load balancer nodes 110store relevant data from the client's SYN packet, and generate a SYN/ACKpacket on behalf of the yet-to-be-chosen server node 130. Referring toFIG. 10C, once the client 160 responds with the first ACK packet in theTCP three-way handshake, the load balancer nodes 110 select a servernode 130, generate an equivalent SYN packet for that server node 130,and attempt to establish an actual TCP connection with the server node130.

Referring again to FIG. 10A, upon receipt of a client SYN packet at theload balancer node 110 serving as the ingress server 112 for the TCPflow, the ingress server 112 extracts the data fields from the SYNpacket and forwards the data to the primary flow tracker 116A for theTCP flow. The primary flow tracker 116A stores the data, for example ina hash table, generates an initial TCP sequence number (for the serverside of the TCP connection), and forwards the same data to the secondaryflow tracker 116B. The secondary flow tracker 116B fabricates a SYN/ACKpacket for the client 160 containing that server TCP sequence number.

In FIG. 10A, the ingress server 112, primary flow tracker 116A, andsecondary flow tracker 116B roles are each performed by different loadbalancer nodes 110. However, in some cases, the load balancer node 110serving as the ingress server 112 for a TCP flow may be the same node110 that serves as the primary flow tracker 116A or the secondary flowtracker 116B for the TCP flow (but not both). The reason that theingress server 112 for a packet flow may be on the same node 110 as aflow tracker 116 for the flow is that the edge router 104 pseudorandomlyselects the ingress server 112 for the flow according to a per-flowhashed multipath routing technique (e.g., an ECMP routing technique),while the flow trackers 116 for the packet flow are determined on aconsistent hash ring according to a consistent hash function applied tothe packet flow's address information. If the ingress server 112 for apacket flow is on the same node 110 as a flow tracker 116 for the packetflow, the data from the SYN packet may only be forwarded from the node110 that implements the ingress server 112 to the other flow tracker 116node 110. For example, in FIG. 10B, the primary flow tracker 116A is onthe same load balancer node 110A as the ingress server 112 for the TCPflow, while the secondary flow tracker 116B is on a different loadbalancer node 110B, and thus the data from the SYN packet is forwardedfrom node 110A (by flow tracker 116A) to the secondary flow tracker 116Bon load balancer node 110B

Referring to FIG. 10C, when non-SYN packets arrive at an ingress server112, the ingress server 112 either knows or does not know which servernode 130 to forward the packets to. The first non-SYN packet to arriveat an ingress server 112 for a TCP flow should be the first TCPacknowledgement (ACK) packet in the TCP three-way handshake (or possiblya subsequent data packet), where the TCP acknowledgement number fieldmatches the server sequence number (+1) that was sent in the SYN/ACKpacket in FIG. 10A. When the ingress server 112 receives a non-SYNpacket for which it has no server mapping, it forwards a message to theprimary flow tracker 116A for the TCP flow, the message includinginformation from the ACK packet such as a sequence number, oralternatively containing the ACK packet itself. In at least some cases,the primary flow tracker 116A remembers the stored data for the TCP flowand confirms that the acknowledged sequence number (+1) matches thevalue that was sent to the client 160 in the SYN/ACK packet. The primaryflow tracker then selects a server node 130 for the TCP flow andforwards another message containing the previously stored data for theTCP flow, the server sequence number, and an IP address for the loadbalancer module 132 on the selected server node 130 to the secondaryflow tracker 116B. The secondary flow tracker 116B confirms the serversequence number, records the information, and sends a fabricated SYNmessage to the load balancer module 132 on the selected server node 130.The TCP flow's CP endpoint pair is now mapped to the load balancermodule 132/server node 130. The load balancer module 132 on the servernode 130 is responsible for creating a legitimate TCP SYN packet for theserver 134 on the server node 130 when it receives the fabricated SYNmessage from the secondary flow tracker 116B. In creating the SYNpacket, the source IP address is populated with the client 160's actualIP address so that the server 134 will believe that it has received adirect TCP connection request from the client 160. The load balancermodule 132 stores the relevant details about the TCP flow, for examplein a local hash table, and sends the TCP SYN packet to the server 134(e.g., injects the SYN packet into the Linux kernel of the server 134).

In FIG. 10C, the ingress server 112, primary flow tracker 116A, andsecondary flow tracker 116B roles are each performed by different loadbalancer nodes 110. However, in some cases, the load balancer node 110serving as the ingress server 112 for a TCP flow will be the same node110 that serves as the primary flow tracker 116A or the secondary flowtracker 116B for the TCP flow (but not both). For example, in FIG. 10D,the secondary flow tracker 116B is on the same load balancer node 110Aas the ingress server 112 for the TCP flow, while the primary flowtracker 116A is on a different load balancer node 110B.

Referring to FIG. 10E, the server 134 (e.g., the Linux kernel) respondswith a SYN/ACK packet that the load balancer module 132 also intercepts.The SYN/ACK packet may contain a different TCP sequence number than wasoriginally delivered to the client 160 in the generated SYN/ACK from thesecondary flow tracker 116B (see FIG. 10A). The load balancer module 132is responsible for applying the sequence number delta to incoming andoutgoing packets. The SYN/ACK packet from the server 134 also triggers amessage (e.g., a UDP message) from the load balancer module 132 back tothe secondary flow tracker 116B to indicate that the connection to theselected server node 130/load balancer module 132/server 134 hassucceeded. Upon receipt of this message, the secondary flow tracker 116Amay record the client and public endpoint pair (CP) mapping between theclient 160 and the server 134 as committed, and send a similar messageto the primary flow tracker 116A which will also record the CP mapping.The primary flow tracker 116A may then forward a CP mapping message tothe ingress server 112, which causes the ingress server 112 to forwardany buffered data packets for the connection to the local load balancermodule 132 on the server node 130 as encapsulated data packets.

Referring to FIG. 10F, the CP mapping for the connection is known to theingress server, so incoming TCP packets received by the ingress server112 for the connection may be encapsulated (e.g., according to UDP) andforwarded directly to the to the local load balancer module 132 on theserver node 130 as encapsulated data packets. The load balancer module132 decapsulates the data packets and sends the TCP packets to theserver 134 on the server node 130, for example by injecting the TCPpackets on to a TCP stack of the kernel. Outbound packets from theserver 134 are intercepted by the load balancer module 132 on the servernode 130, encapsulated (e.g., according to UDP), and forwarded to anarbitrary load balancer node 110 that the load balancer module 132randomly selects as the egress server 114 for this connection. Theegress server 114 decapsulates the packets and sends the decapsulatedpackets to the client 116. The egress function of the selected loadbalancer node 110 is stateless, so a different load balancer node 110can be selected as the egress server 114 for the connection in the eventof failure of the load balancer node 110 serving as the egress server.However, generally the same load balancer node 110 is used as the egressserver 114 for the duration of the connection to reduce or eliminatere-ordering of the outbound packets.

Referring to FIG. 10G, in at least some embodiments, if the loadbalancer module 132A on a server node 130A that is selected by theprimary flow tracker 116A (see FIG. 10C) determines that it isoverloaded, it has the option to reject the fabricated SYN messagereceived from the secondary flow tracker 116B (see FIG. 10C). In atleast some embodiments, the fabricated SYN message includes a time tolive (TTL) value or counter that allows for a maximum number ofrejections. In at least some embodiments, if this TTL value reacheszero, the load balancer module 132A may either accept the connection ordrop the connection to shed load. If the load balancer module 132Adecides to reject the connection, it decrements the TTL value and sendsa reject message to the secondary flow tracker 116B. The secondary flowtracker 116B resets the CP mapping and sends a release message to theprimary flow tracker 116A to do the same. The primary flow tracker 116Achooses a new load balancer module 132B on another server node 130B andsends a new target message back to the secondary flow tracker 116B,which sends a new fabricated SYN message to the newly chosen loadbalancer module 132B. Note that packet drops may result in this sequencefailing to complete; however, a retransmission from the client 160 maytrigger the load balancer module selection process again at the primaryflow tracker 116A, which may, but does not necessarily, choose the sameload balancer module 132 for the connection if it has not learned aboutthe previous rejection of the fabricated SYN packet.

In at least some embodiments, the TTL counter may be used to preventcontinuously sending connection requests to server nodes 130, which mayoccur for example if all the server nodes 130 are busy. In at least someembodiments, each time a load balancer module 132 rejects a connectionrequest on behalf of a respective server node 130, the load balancermodule 132 decrements the TTL counter. The flow tracker nodes 116 maymonitor the TTL counter and, as long as the TTL counter is not zero (oris above some specified threshold), may select another server node 130and try again. If the TTL counter reaches zero (or reaches the specifiedthreshold), the connection request is dropped and no further attemptsare made by the flow tracker nodes 116 to send a connection request to aselected one of the server nodes 130 for that connection. In at leastsome embodiments, an error message may be sent to the respective client160.

In at least some embodiments, the distributed load balancer systemsupports multiple public IP addresses. As such, it is possible that aclient 160 may initiate two TCP connections from the same client portnumber to two different public IP addresses. These TCP connections aredistinct from the client 160's point of view, but internally thedistributed load balancer may map the connections to the same servernode 130, which would result in a collision. In at least someembodiments, to detect and handle possible collisions, the load balancermodule 132, upon receiving the fabricated SYN packet from the secondaryflow tracker 116B as shown in FIGS. 10C and 10D, may compare the addressinformation to its active connections and, if this connection wouldcause a collision, reject the connection request as shown in FIG. 10G.

Handling Load Balancer Node Failures and Additions

In many conventional load balancers, some or all existing connectionsare lost in the event of a load balancer failure. In at least someembodiments, in the event of failure of a single load balancer node 110,the distributed load balancing system may maintain at least some of theestablished connections so that the clients and servers can continue toexchange packets via the connections until the connections completenormally. In addition, the distributed load balancing system maycontinue to service connections that were in the process of beingestablished at the time of failure.

In at least some embodiments of the distributed load balancing system, afailure recovery protocol may be implemented that may recover existingclient connections in the event of a single load balancer node 110failure. Multiple load balancer node 110 failures, however, may resultin lost client connections. In at least some embodiments, TCPretransmissions between a client 160 and a server 134 may be used as ameans of recovery following a load balancer node 110 failure.

In addition to potential load balancer node 110 failures, new loadbalancer nodes 110 may be added to the distributed load balancer system.These new nodes 110 may be added to the load balancer layer and thus tothe consistent hash ring, and load balancer node 110 roles regardingexisting client connections may be adjusted according to the change, asnecessary.

Handling Flow Tracker Node Failures and Additions

In at least some embodiments, as each connection is established (see,e.g., FIGS. 10A through 10G), the connection state information is passedthrough two load balancer nodes 110, referred to as the primary andsecondary flow trackers, which may be determined using a consistent hashalgorithm that, for example, uses the (client IP:port, public IP:port)tuple as hash function input. In the event of a single load balancernode 110 failure, at least one of the surviving load balancer nodes 110may continue to be mapped via the consistent hash function and maycontain the necessary state information for a connection to directpackets to the selected server node 130 for a connection. In addition,in the case of an addition of a load balancer node 110 to the consistenthash ring, state information for connections may be refreshed to theappropriate flow trackers.

FIGS. 11A through 11D illustrate the handling of events that effectmembership in the load balancer node consistent hash ring, according toat least some embodiments. These events may include, but are not limitedto, adding a new primary flow tracker node, adding a new secondary flowtracker node, failure of a primary flow tracker node, and failure of asecondary flow tracker node.

FIG. 11A illustrates handling the addition of a new primary flow trackernode to the consistent hash ring. The top row of FIG. 11A shows flowtracker 116A as the primary flow tracker for one or more clientconnections and flow tracker node 116B as the secondary flow tracker forthe same connection(s). In the bottom row of FIG. 11A, a new flowtracker node 116C has been added, and becomes the primary flow trackerfor the client connection(s). Flow tracker node 116A, formerly theprimary flow tracker, becomes the secondary flow tracker, while flowtracker node 116B, formerly the secondary flow tracker, becomes a nextflow tracker in the consistent hash ring. State information for theclient connection(s) that was maintained by flow trackers 116A and 116Bmay be provided to the new primary flow tracker 116C. In addition, flowtracker 116B may “forget” its formerly tracked connections in the roleof secondary flow tracker.

FIG. 11B illustrates handling the addition of a new secondary flowtracker node to the consistent hash ring. The top row of FIG. 11B showsflow tracker 116A as the primary flow tracker for one or more clientconnections and flow tracker node 116B as the secondary flow tracker forthe same connection(s). In the bottom row of FIG. 11B, a new flowtracker node 116C has been added, and becomes the secondary flow trackerfor the client connection(s). Flow tracker node 116A remains as theprimary flow tracker for the connection(s), while flow tracker node116B, formerly the secondary flow tracker, becomes a next flow trackerin the consistent hash ring. State information for the clientconnection(s) that was maintained by flow trackers 116A and 116B may beprovided to the new secondary flow tracker 116C. In addition, flowtracker 116B may “forget” its formerly tracked connections in the roleof secondary flow tracker.

FIG. 11C illustrates handling the failure of a primary flow tracker nodein the consistent hash ring. The top row of FIG. 11C shows flow tracker116A as the primary flow tracker for one or more client connections,flow tracker node 116B as the secondary flow tracker for the sameconnection(s), and flow tracker node 116C as a next flow tracker in theconsistent hash ring. In the bottom row of FIG. 11C, primary flowtracker node 116A has failed. Flow tracker node 116B becomes the primaryflow tracker for the connection(s), while flow tracker node 116C becomesthe secondary flow tracker for the connection(s). State information forthe client connection(s) is maintained by flow tracker 116B and may beprovided to the new secondary flow tracker 116C.

FIG. 11D illustrates handling the failure of a secondary flow trackernode in the consistent hash ring. The top row of FIG. 11D shows flowtracker 116A as the primary flow tracker for one or more clientconnections, flow tracker node 116B as the secondary flow tracker forthe same connection(s), and flow tracker node 116C as a next flowtracker in the consistent hash ring. In the bottom row of FIG. 11D,secondary flow tracker node 116B has failed. Flow tracker node 116Aremains as the primary flow tracker for the connection(s), while flowtracker node 116C becomes the secondary flow tracker for theconnection(s). State information for the client connection(s) ismaintained by flow tracker 116B and may be provided to the new secondaryflow tracker 116C.

In at least some embodiments, the load balancer modules 132 on theserver nodes 130 perform connection publishing to the load balancernodes 110. In at least some embodiments, the connection publishingperiodically (e.g., once a second) or aperiodically pushes currentconnection state information from the server nodes 130 to the loadbalancer nodes 110 serving as flow tracker nodes and ingress nodes,which acts to refresh or restore the connection mappings to both theprimary and secondary flow tracker nodes for the connections. In atleast some embodiments, a load balancer module 132 may detect a flowtracker membership change, for example as illustrated in FIGS. 11Athrough 11D. In response, the load balancer module 132 may perform aconnection publication to populate the state information for theconnections in the primary and secondary flow tracker nodes, which mayhave changed for the connections when the membership changed. Note thatconnection publishing may allow at least some established connections tobe recovered in the event of multiple load balancer node failures.

Failure-Related Message Flow

In at least some embodiments, the protocol between the primary andsecondary flow tracker nodes may include a correction or synchronizationfunctionality. For example, referring to FIG. 11A, when a new primaryflow tracker node 116C joins the consistent hash ring, the new node 116Cmay lay claim to the consistent hash keyspace for some number ofconnections (˜1/N) and begin receiving traffic related to theseconnections from the edge router 104. However, the new primary flowtracker node 116C does not have any state stored for the connections, soit may operate on each packet as if it were the first packet receivedfrom the client 160. The primary flow tracker is responsible forgenerating server TCP sequence numbers in response to SYN packets (see,e.g., FIG. 10A) and for selecting server nodes 130 in response to thefirst ACK packet from a client 160 (see, e.g., FIG. 1), and thesegenerated values may disagree with values chosen by the previous primaryflow tracker (flow tracker node 116A in FIG. 11A). However, in at leastsome embodiments the consistent hash algorithm assigns the previousprimary flow tracker (flow tracker node 116A in FIG. 11A) into thesecondary flow tracker role, and this flow tracker still retains thepreviously stored state for the connections. Thus, in at least someembodiments, when the secondary flow tracker (flow tracker node 116A inFIG. 11A) detects a discrepancy in information received from the primaryflow tracker 116C, it can send update messages back to the primary flowtracker 116C to bring the two load balancer nodes 110 serving as flowtrackers for the connections into synchronization. Similar methods maybe used to synchronize the flow trackers after other changes in theconsistent hash ring membership.

Load Balancer Module Details

In at least some embodiments, the load balancer module 132 is acomponent of the distributed load balancer system that resides on eachof the server nodes 130. Roles of the load balancer node 132 include,but are not limited to, decapsulating packets received from the loadbalancer nodes 110 and sending the decapsulated packets to the server134 on the server node 130, and encapsulating outgoing packets from theserver 134 and sending the encapsulated packets to a load balancer node110.

In at least some embodiments, incoming packets to the load balancermodules 132 on the server nodes 130 from the load balancer nodes 110serving as ingress servers 112 are stateless protocol (e.g., UDP)packets that encapsulate the actual client data packets. Eachencapsulated client data packet has the original clientIP:port of arespective client 160 as the source address and the server 134publicIP:port as the destination address. The load balancer modules 132strip the encapsulation from the client data packets and send thepackets to the respective servers 134 on the server nodes 130, forexample by redirecting the packets to a localhost TCP flow.

In at least some embodiments, outgoing packets from the servers 134 tothe load balancer nodes 110 serving as egress servers 114 are statelessprotocol (e.g., UDP) packets that encapsulate the outgoing IP packets.The load balancer modules 132 encapsulate the outgoing IP packets andsend the encapsulated packets to the egress servers 114 via the fabric120. Each encapsulated outgoing IP packet has the server 134publicIP:port as the source address and the clientIP:port of arespective client 160 as the destination address.

Load Balancer Module Functionality

In at least some embodiments, functions of the load balancer module 132on a server node 130 may include one or more of, but are not limited to:

-   -   Terminating UDP tunnels from the load balancer node(s) 110, e.g.        from the ingress server 112 handling a connection to a client        160. This includes stripping UDP encapsulation from incoming        client data packets received from the ingress servers 112.    -   Selecting an egress server 114 to receive outgoing traffic for a        connection.    -   Intercepting outgoing IP packets on a connection to the        respective server 134, encapsulating the outgoing IP packets for        the connection, and sending the encapsulated packets to the        egress server 114.    -   Mangling the sequence number in incoming and outgoing packets so        that the sequence number aligns with the sequence number        generated by the flow tracker nodes 116 when the flow tracker        nodes 116 sent a SYN/ACK to the client 160.    -   Making the decision on whether to accept or reject a connection        for the respective server 134, for example based on one or more        metrics indicating the respective server 134's current load.    -   Detecting and rejecting connections from the same clientIP:port        address to the respective server 134 if there is an active        connection for that clientIP:port address to avoid collisions.    -   Connection tracking and connection publishing.

Load Balancer Module Configuration Information

In at least some embodiments, each load balancer module 132 may acquireand locally store one or more of, but is not limited to, the followingsets of information for its configuration: a set of load balancer node110 endpoints; a set of valid public IP addresses which it is to serve;and the port number(s) on which the respective server 134 acceptsincoming connections. In a least some embodiments, this information maybe acquired from or updated by accessing or querying a configurationservice 122 component of the distributed load balancer system, asillustrated in FIG. 1. Other methods of acquiring the information may beused in some embodiments.

Load Balancer Module Packet Handling

The following describes load balancer module 132 operations for inboundtraffic and outbound traffic according to at least some embodiments. Inat least some embodiments, when an inbound data packet is received bythe load balancer module 132, the data packet is decapsulated from theUDP packet, and the destination address in the decapsulated TCP packetis first validated against a set of configured valid public IPaddresses. If there is no match, the packet is dropped or ignored. In atleast some embodiments, the load balancer module 132 may adjust thesequence number in the TCP header by a constant delta so that thesequence number matches the randomly chosen sequence number generated bythe flow tracker nodes 116 that sent the SYN/ACK packet to the client160. The load balancer module 132 records the mapping from the[Client:Public] endpoint to the [Client:Server] endpoint as an internalstate.

In at least some embodiments, for outbound TCP packets from the server134, the load balancer module 132 first checks its internal state todetermine if the packet is for an active connection that the loadbalancer module is managing. If it is not, the load balancer module 132just passes the packet through. If it is, the load balancer module 132encapsulates the outgoing TCP packet, for example according to UDP, andforwards the encapsulated packet to a load balancer node 110 that wasselected as the egress server 114 for this connection. In at least someembodiments, the load balancer module 134 may adjust the TCP sequencenumber in the outgoing TCP packet by a constant delta so that it alignswith the sequence number generated by the flow tracker nodes 116 thatsent the SYN/ACK packet to the client 160.

Connection Tracking

In at least some embodiments, the load balancer module 132 on eachserver node 130 manages a hash table containing connection details forevery active client connection to the respective server 134. In at leastsome embodiments, the key for the hash table is the (clientIp:port,publicIp:port) tuple. In at least some embodiments, the connection statefor each client connection may include one or more of, but is notlimited to:

-   -   The client IP:Port    -   The public IP:Port    -   The initial server TCP sequence number provided by the flow        tracker 116 nodes.    -   The server TCP sequence number delta.    -   The original primary flow tracker IP address.    -   The original secondary flow tracker IP address.    -   The IP address of the last detected ingress server 112.    -   An expiration time for this entry    -   Least Recently Used (LRU)/Collision indices.

In at least some embodiments, each load balancer module 132 periodicallygenerates connection publishing messages to the primary and secondaryflow tracker nodes for all active client connections. In at least someembodiments, the content of /proc/net/tcp is scanned and intersectedwith the active connections in the load balancer module's hash table sothat they will continue to be published to the flow tracker nodes untilthe Linux kernel stops tracking the connection. Connection publishingwill be discussed in more detail later in this document.

Sequence Number Mangling

As previously described, in at least some embodiments the load balancernodes 110 generate SYN/ACK packets in response to client 160 SYN packetson behalf of the server 134. Only after the client 160 sends an ACKpacket (the TCP three-way handshake) does a load balancer module 110send any data to a load balancer module 132 on a server node 130. Whenthe load balancer module 132 is first instructed to establish a clientconnection, the load balancer module 132 locally fabricates a SYN packetto begin a TCP connection with the server 134 on the server node 130,and intercepts the server 134's corresponding SYN/ACK packet. Typically,the server 134 (e.g., the Linux kernel on the server node 130) selectsan entirely different TCP sequence number than the one the clientreceived in the SYN/ACK packet from the load balancer nodes 110. Thus,in at least some embodiments, the load balancer module 132 may correctfor the sequence numbers in all the packets in the TCP connectionbetween the client 160 and the server 134. In at least some embodiments,the load balancer module 132 computes the difference between thesequence number generated by the load balancer nodes 110 and thesequence number generated by the server 134 and stores the difference asa delta value in the hash table entry for the TCP connection. Whenincoming data packets arrive from the client 160 on the connection, theTCP header will contain acknowledgement numbers that will not align withthe sequence number used by the server 134, so the load balancer module132 subtracts the delta value (e.g., using two's complement) from thesequence number value in the TCP header. The load balancer module alsoadds the delta value to the sequence number in outbound packets from theserver 134 to the client 130 on the connection.

Health Checking in the Distributed Load Balancer System

In at least some embodiments of the distributed load balancer system,each load balancer node 110 requires a consistent view of the healthymembers in the load balancer implementation (i.e., of the healthy loadbalancer nodes 110 and server nodes 130) for at least the followingreasons:

-   -   Load balancing—The load balancer nodes 110 need to detect server        node 130 failures and converge on a set of healthy server nodes        130 that can accept client traffic.    -   Distributed state management—The load balancer is a distributed        system with the state shared/replicated across multiple load        balancer nodes 110 (e.g., according to a consistent hashing        mechanism). In order to properly handle client traffic, each        load balancer node 110 needs to have an eventually consistent        view of the healthy member nodes 110 in the load balancer        implementation.

To accomplish this, at least some embodiments of the distributed loadbalancer system may implement embodiments of a health check protocolthat monitors nodes in the load balancer implementation and detectsunhealthy nodes as soon as possible. The health check protocol maypropagate health information among the nodes in the load balancerimplementation, and may provide methods that enable the nodes toconverge on a set of healthy nodes. In addition, the health checkprotocol may provide mechanisms for reporting healthy/unhealthy nodesand state changes in the load balancer implementation.

In at least some embodiments, the health check protocol may be based onone or more of, but not limited to, the following assumptions:

-   -   All nodes in the load balancer implementation are known. (I.e.,        the health check protocol may not perform discovery).    -   All node failures are fail-stop.    -   All messages between nodes are stateless protocol (e.g., UDP)        messages, and the messages may be dropped, delayed, duplicated        or corrupted. There are no guarantees on message delivery.

In at least some embodiments, a node in a load balancer implementation(e.g., a load balancer node 110 or server node 130) may be consideredhealthy under the following conditions:

-   -   All of the node's internal components are in ready state (ready        to handle client traffic).    -   The node's incoming/outgoing network links are healthy (for at        least the network interface controllers (NICs) on which client        traffic flows).

FIG. 12 is a high-level flowchart of a health check method that may beperformed by each load balancer node according to a health checkinterval, according to at least some embodiments. As indicated at 1000,at each load balancer interval, for example every 100 milliseconds, eachload balancer (LB) node 110 may health check at least one other LB node110 and at least one server node 130. As indicated at 1002, the loadbalancer node 110 may update its locally stored health informationaccording to the health checks. As indicated at 1004, the load balancernode 110 may then randomly select at least one other load balancer node110 and send its health information to the selected load balancernode(s) 110. In at least some embodiments, the node 110 may also send alist of healthy load balancer nodes 110 to one or more server nodes 130,for example to the same server node(s) 130 that are health checked bythe node 110. The elements of FIG. 12 are explained in more detail inthe following discussion.

In at least some embodiments of the health check protocol, a loadbalancer node 110 does not assert its own health to the other loadbalancer nodes 110. Instead, one or more other load balancer nodes 110may health-check the node 110. For example, in at least someembodiments, each load balancer node 110 may periodically oraperiodically randomly select one or more other nodes 110 tohealth-check. As another example, in at least some embodiments, one ormore other load balancer nodes 110, for example the two nearestneighbors of a given load balancer node 110 on an ordered list of nodes110 such as a consistent hash ring, may each periodically oraperiodically check the health of the given node 110. In at least someembodiments, health checking a node 110 may include using health pingssent to the NICs 1114 on the node 110 as illustrated in FIG. 23. In atleast some embodiments, if a first node 110 determines that a secondnode 110 is healthy via a health check, the first node 110 may update(e.g., increment) the heartbeat counter for the second node 110 storedin local health information for the load balancer nodes 110. The firstnode 110 periodically or aperiodically sends its local healthinformation to one or more other load balancer nodes 110 in the loadbalancer implementation, which may update their own local healthinformation accordingly (e.g., by incrementing the heartbeat counter forthe second node) and send their updated local health information to oneor more other nodes 110. The heartbeat information for the second node110 may thus be propagated to the other nodes 110 in the load balancerimplementation. As long as the second node 110 is healthy, all of theother nodes 110 that are reachable from the second node 110 should thussee the second node 110's heartbeat counter getting incremented on aconsistent basis, e.g. once a second or once every ten seconds. If thesecond node 110 is detected to be unhealthy by the node(s) 110 thatcheck its health, no heartbeat for the node 110 is sent by the healthchecking nodes 110 and, after some time threshold, the other nodes 110in the load balancer implementation 110 consider the node 110 inquestion to be unhealthy, or down.

In at least some embodiments, a load balancer node 110 may check one ormore aspects of its own internal state and, if the node 110 detects thatit is unhealthy for some reason, the node 110 may stop responding to thehealth pings from other nodes 110 that check its health. Thus, the nodes110 checking the unhealthy node 110's health may consider the node 110as unhealthy, and may not propagate heartbeat increments on behalf ofthe node 110.

Health Check Protocol Details

In at least some embodiments, the health check protocol may leverage aheartbeat counter technique and gossip protocol technology. The healthcheck protocol may be considered to have two main parts—health checkingand gossip/failure detection.

Health checking—Every load balancer node 110 in the load balancerimplementation may periodically or aperiodically health check one ormore other nodes 110 in the implementation. Methods by which the one ormore other nodes are determined are discussed later. A core idea ofhealth checking is that if a node 110 health checks another node 110 anddetermines that the other node 110 is healthy, the checking node 110asserts that the other node 110 is healthy by incrementing andpropagating a heartbeat counter for the other node 110. In other words,the nodes 110 do not assert their own health to the other nodes;instead, one or more other nodes 110 check and assert the health of eachnode 110 in the load balancer implementation.

Gossip/failure detection—In at least some embodiments, the health checkprotocol may leverage a gossip protocol to propagate load balancer node110 health information among the member load balancer nodes 110 in theload balancer implementation. The gossip protocol converges rapidly, andprovides eventual consistency guarantees that are sufficient for thepurposes of the distributed load balancing system. In at least someembodiments, using the gossip protocol, each load balancer node 110maintains a heartbeat counter for each other node 110 in the loadbalancer implementation, for example in a heartbeat list. Each loadbalancer node 110 periodically or aperiodically performs a health checkof at least one other load balancer node 110 as described above, andincrements the heartbeat counter for a node 110 upon determining via thehealth check that the checked node 110 is healthy. In at least someembodiments, each load balancer node 110 periodically or aperiodicallyrandomly selects at least one other node 110 in the load balancerimplementation to which it sends its current heartbeat list. Uponreceipt of a heartbeat list from another node 110, a load balancer node110 merges the heartbeat information in the received list with its ownheartbeat list by determining the maximum heartbeat counter for eachnode 110 in the two lists (the received lists and its own list) andusing the determined maximum heartbeat counter in its own heartbeatlist. In turn, this heartbeat list gets sent to another randomlyselected node 110, which updates its own heartbeat list accordingly, andso on. Using this technique, heartbeat information for each healthy node110 is eventually (e.g., in a few seconds) propagated to all of theother load balancer nodes 110 in the load balancer implementation. Aslong as the heartbeat counter keeps increasing for a given load balancernode 110, it is considered to be healthy by the other nodes 110. If aload balancer node 110's heartbeat counter does not get incremented fora specified period by the health checking and gossiping method, thenother load balancer nodes 110 may converge on the load balancer node 110being considered unhealthy.

Health Checking Load Balancer Nodes

The following describes a method for health checking a load balancernode 110 that may be performed by another load balancer node 110,according to at least some embodiments. With reference to FIG. 23, in atleast some embodiments, a load balancer node 110 may be consideredhealthy if one or more of the following conditions are determined forthe node 110:

-   -   The processor threads (e.g., core packet processing code 1108        threads) of the node 110 are in the ready state (internal).    -   The node 110 knows the edge router 104's IP address and/or MAC        address (internal).    -   All of the threads and/or protocol handlers of the node 110 are        in the ready state (internal).    -   The incoming and outgoing links from the north side (edge router        104/border network) and from the south side (servers        130/production network) are active (external).    -   The node 110 can receive and dispatch packets via the network        interface controllers (NICs) used in the load balancer        implementation. For example, in an example load balancer node        110 embodiment as shown in FIG. 23, the node 110 should        successfully receive and dispatch packets via the north-facing        NIC 1114A and the south-facing NIC 1114B.

If one or more of these health conditions does not hold for a given node110, the node 110 may be considered not healthy. Note that, in someembodiments, a node 110 is only considered healthy if all of the aboveconditions hold for the node 110.

In at least some embodiments, in addition to the above healthconditions, a third NIC, shown in FIG. 23 as NIC 1114C, on each loadbalancer node 110 that may, for example, be used for control planecommunications may also be checked by a health-checking node 110 bysending packets to and receiving packets from the NIC and, if the checkof the third NIC fails, the node 110 being checked may be consideredunhealthy.

FIG. 13 illustrates an example method for health checking a loadbalancer node from another load balancer node, according to at leastsome embodiments. In this example, load balancer node 110A is healthchecking load balancer node 110B. Each node 110A and 110B has anorth-facing NIC (NIC 1114A in FIG. 23) and a south-facing NIC (NIC1114B in FIG. 23). At 1, node 110A sends a packet (e.g., a ping packet)from its north-facing NIC to the north-facing NIC of node 110B via edgerouter 104. Node 110B receives the packet on its north-facing NIC, andat 2 sends a response from its north-facing NIC to the north-facing NICof node 110A via fabric 120, provided the conditions given in the listabove are satisfied. After receiving the response on its north-facingNIC, at 3, node 110A sends a packet (e.g., a ping packet) from itssouth-facing NIC to the south-facing NIC of node 110B via fabric 120.Node 110B receives the packet on its south-facing NIC, and at 4 sends aresponse from its south-facing NIC to the south-facing NIC of node 110Avia edge router 104, provided the conditions given in the list above aresatisfied. Upon receiving the response on its south-facing NIC, node110A considers node 110B to be healthy and increments node 110B's localheartbeat counter, which may then be propagated to other nodes 110according to a gossip protocol as previously described.

As an alternative to the above, in some embodiments, load balancer node110B may respond to the first ping message, received at its north-facingNIC, via its south-facing NIC to the south-facing NIC of node 110A, andrespond to the second ping message, received at its south-facing NIC,via its north-facing NIC to the north-facing NIC of node 110A.

In addition, in some embodiments, node 110A may also health check athird NIC of node 110B that is used for control plane communications(shown as NIC 1114C in FIG. 23) by pinging node 110B's third NIC fromits own third NIC and receiving a response to the ping message on itsthird NIC from node 110B's third NIC if node 110B is healthy. The pingmessage and response may pass through one or more control planedevice(s) 170, for example a network switch.

The above-described health check mechanism exercises all of the incomingand outgoing links and data paths of node 110B in all directions (north,south, and through the control plane) as well as all of node 110B'sNICs, and also verifies the internal health of node 110B as the pingpackets traverse the internal queues and dispatching of node 110B aswould a client packet.

Assigning Health Checking Responsibilities to Load Balancer Nodes

In at least some embodiments, every load balancer node 110 in a loadbalancer implementation has access to a list (e.g., a sorted list) ofall of the other load balancer nodes 110 in the load balancerimplementation, for example via a configuration function and/or via aconfiguration service 122 component as shown in FIG. 1. In at least someembodiments, each load balancer node 110 may randomly select one or moreother nodes 110 on the list to health check at each health checkinterval, incrementing their heartbeat counter if determined healthy.Note that the list includes all load balancer nodes 110 in the loadbalancer implementation whether currently considered healthy orunhealthy via the health check mechanism, and currently unhealthy nodes110 may be randomly selected from the list and health checked as well ashealthy nodes 110. Thus, a currently unhealthy node 110 may bedetermined to be healthy by one or more nodes 110 that health check thenode 110, its heartbeat counter may be incremented and propagated to theother nodes 110, and the unhealthy node 110 may thus return to healthystatus.

Alternatively, in some embodiments, each load balancer node 110 mayassume responsibility for health checking one or more other nodes 110 inthe list and incrementing their heartbeat counter if determined healthy.For example, in some embodiments, each node 110 may assumeresponsibility for two other nodes, for example its “left” (or previous)and “right” (or next) nearest neighbor nodes 110 in the list. Note thatthe list may be considered circular and a node 110 at the “end” of thelist may assume responsibility for health checking a node 110 at the“beginning” of the list, and vice versa. In some embodiments, the twoother nodes 110 may be otherwise selected, for example as the twonearest neighbors next on the list. In some embodiments, each node 110may assume responsibility for health checking more than two other nodes110 on the list, for example three or four other nodes 110. In at leastsome embodiments, if a neighbor node 110 that is being checked by a node110 is determined to be unhealthy, then the node 110 may assumeresponsibility for health checking at least one node on the list thatthe unhealthy neighbor node 110 was responsible for checking. In atleast some embodiments, in addition to health checking its neighbornodes 110 (e.g., a “left” and “right” neighbor node), each load balancernode 110 may also periodically or aperiodically randomly select a node110 in the ring and perform a health check of that randomly selectednode 110 and, if healthy, increment and propagate the random node 110'sheartbeat. In at least some embodiments, all other nodes 110 in theordered list are considered for the random selection and health checkregardless of whether the other node 110 was previously consideredhealthy or not.

In at least some embodiments, each node 110 performs the health check ofone or more randomly selected nodes 110, or alternatively of itsneighbor nodes 110 and a randomly selected node, at a regular interval,which may be referred to as the health check interval. For example, insome embodiments, the heartbeat interval may be 100 milliseconds,although shorter or longer intervals may be used. In addition, in atleast some embodiments, each node 110 sends or “gossips” its currentheartbeat list to at least one other randomly selected node 110 at aregular interval, which may be referred to as a gossip interval. In someembodiments, the health check interval and the gossip interval may bethe same, although they are not necessarily the same.

FIG. 14 graphically illustrates a load balancer node health checking oneor more other load balancer nodes, according to at least someembodiments. In this example, there are eight load balancer nodes110A-110H in the load balancer implementation. The dotted circlerepresents an ordered list of all nodes 110 in the implementation. Insome embodiments, each node 110 may randomly select one or more othernodes 110 on the list to health check at each interval. As analternative, in some embodiments, each load balancer node 110 may assumeresponsibility for checking one or more particular nodes 110 on theordered list, for example node 110A may take responsibility forhealth-checking its two nearest neighbor nodes 110B and 110H accordingto the ordered list as shown in FIG. 14. In addition, the load balancernode may also randomly select another node 110 from the ordered list ateach health check interval. As shown in this example, node 110A has alsorandomly selected node 110F to health check. At the gossip interval,node 110A randomly selects some other healthy node 110, for example node110D, and sends its current heartbeat list to the selected other node110, for example in a UDP message. A node 110, upon receiving aheartbeat list from another node 110, may update its own heartbeat listaccordingly and propagate the heartbeat list to one or more randomlyselected nodes 110 at the next gossip interval.

Health Checking the Server Nodes

In addition to health checking the load balancer nodes 110 as describedabove, embodiments of the health check protocol may perform healthchecking of the server nodes 130 including the load balancer modules 132and servers 134 on those nodes 130. In at least some embodiments, aserver node 130 may be considered healthy if one or both of thefollowing conditions are determined for the node 130:

-   -   The load balancer module 132 is healthy.    -   The server node 130 responds successfully to health pings (e.g.,        L7 health pings).

FIG. 15 illustrates the load balancer nodes health checking the servernodes, according to at least some embodiments. In at least someembodiments, every load balancer node 110 in a load balancerimplementation has access to a list of all of the other load balancernodes 110 in the load balancer implementation, as well as a list of allserver nodes 130 in the load balancer implementation. The list(s) may beobtained and updated, for example via a configuration function and/orvia a configuration service 122 component as shown in FIG. 1. In atleast some embodiments, the server nodes 130 may be consistent hashedagainst the healthy load balancer nodes 110 to form a consistent hashring as illustrated in FIG. 15. In at least some embodiments, eachserver node 130 in the ring is health checked by two healthy loadbalancer nodes 110 in the ring. For example, in FIG. 15, server node130A is health checked by load balancer nodes 110A and 110C. These twonodes 110 may be referred to as the first (node 110A) and second (node110B) health checking nodes 110 for the server node 130 in theconsistent hash ring. Note that a given healthy load balancer node 110may health check more than one server node 130. For example, in FIG. 15,load balancer node 110A also health checks server nodes 130B and 130C.In addition, a given node balancer node 110 may be a first healthchecking node 110 for one or more server nodes 130 and a second healthchecking node 110 for one or more other server nodes 130. For example,in FIG. 15, load balancer node 110A is the first health checker node forserver nodes 130A and 130B and the second health checker node for servernodes 130C and 130D.

In at least some embodiments, if a load balancer node 110 fails, themembership in the consistent hash ring changes, and one or more othersof the load balancer nodes 110 that are still healthy and thus still onthe consistent hash ring may assume responsibility for health checkingthe server nodes 130 previously health checked by the failed node 110.

In at least some embodiments, each healthy node 110 performs the healthcheck of its assigned server nodes 130 at a regular interval, which maybe referred to as a server check interval. In at least some embodiments,the server check interval may be greater than or equal to the gossipinterval previously mentioned.

In at least some embodiments, to perform a health check of a server node130, a healthy load balancer node 110 (e.g., node 110A in FIG. 15)initiates a health ping message (e.g., a L7 HTTP health ping message) toa server node 130 (e.g., server node 130A in FIG. 15). If healthy, theserver node 130 sends a ping response back to the load balancer node110. In at least some embodiments, the ping message is received andprocessed by the load balancer module 132 on the server node 130, so thehealth check ping, if successful, establishes that the module 132 on theserver node 130 is healthy. Upon receiving the response to the ping, theload balancer node 110 considers the server node 130 as healthy, andincrements a heartbeat counter for the server node 130.

In at least some embodiments, the heartbeat counters for all servernodes 130 health checked by a given healthy load balancer node 110 maybe propagated to the other load balancer nodes 110, for exampleaccording to the gossip technique previously described for the loadbalancer node 110 heartbeat counters in which each node 110 sends itsheartbeat list to at least one other randomly selected node 110 at aregular interval (the gossip interval), and the receiving node 110updates its own heartbeat list according to the maximum values in thetwo lists.

Failure Detection and Gossip

In at least some embodiments, the information obtained through the loadbalancer node 110 health checks and the server node 130 health checksdescribed above may need to be propagated to all the nodes 110 in theload balancer implementation so that all load balancer nodes 110 canmaintain consistent view of the load balancer implementation. Asdescribed above, in at least some embodiments, the load balancer nodes110 may communicate with each other according to a gossip protocol toexchange and propagate this health information and to detect loadbalancer node 110 and server node 130 failures.

In at least some embodiments, at a regular interval (referred to as thegossip interval), each load balancer node 110 randomly selects anotherload balancer node 110 and sends the other node 110 its view of healthyload balancer nodes 110 and server nodes 130 along with the heartbeatcounters for the load balancer nodes 110 and server nodes 130. As longas a load balancer node or server node 130 is healthy, the node willpass its health checks and its heartbeat counter will keep increasing.If the heartbeat counter for a node does not change for a specifiedinterval (which may be referred to as a fail time interval), then thenode is suspected to have failed by the load balancer nodes 110. Once anode is suspected to have failed, the load balancer nodes 110 may waitfor a specified interval (which may be referred to as the unhealthy timeinterval) before determining that the node is unhealthy. This unhealthytime interval allows the load balancer nodes 110 to wait until all theload balancer nodes 110 learn that the node has failed.

FIG. 16 graphically illustrates a state for, or view of, health ofanother node (either a load balancer node 110 or server node 130) thatmay be maintained by a load balancer node 110, according to at leastsome embodiments. Assume the load balancer node 110 starts with a viewof the node in question as being healthy, as indicated at 300. Thisindicates that the heartbeat counter for the node has been incrementing.However, if the node's heartbeat counter does not increase for aspecified interval (the fail time interval) as indicated at 302, thenthe load balancer node 110 suspects that the node has failed, asindicated at 304. If the node's heartbeat counter does not increase fora specified interval (the unhealthy time interval) as indicated at 306,then the load balancer node 110 considers the node unhealthy, asindicated at 308. However, if the heartbeat counter for the nodeincrements before the unhealthy time interval expires as indicated at310, the load balancer node 110 again considers the node as healthy 300.Similarly, receiving a heartbeat increment for an unhealthy node asindicated at 312 can cause the node to be considered as healthy 300.

Determining that a node is unhealthy may involve different actions bythe load balancer node(s) 110 depending on whether the unhealthy node isa load balancer node 110 or a server node 130, and also depending on theload balancer node's 110 relationship with the unhealthy node, asdescribed elsewhere herein.

Load Balancer Node Data

In at least some embodiments, each load balancer node 110 may maintaindata about the state of the load balancer implementation. In at leastsome embodiments, this data may be maintained in one or more datastructures on each load balancer node 110 including but not limited to ahealthy load balancer node list, a suspect load balancer node list, anda heartbeat list. FIG. 17 illustrates an example load balancer node 110that maintains a healthy load balancer node list 320, a suspect loadbalancer node list 322, an unhealthy load balancer node list 324, and aload balancer node heartbeat list 326.

In at least some embodiments, each load balancer node 110 may maintain ahealthy load balancer node list 320, which is a list of healthy loadbalancer nodes 110 that may, for example, be used to determine whichnodes 110 are healthy and are thus participating in the gossip protocol.Only the nodes 110 on the list 320 are involved in the propagation ofload balancer information via the gossip protocol, only the nodes 110 onthe list 320 are considered to be in the consistent hash ring, and onlythe nodes 110 on this list health-check server nodes 130. A node 110 mayrandomly select another node 110 from this list 320 to which itsheartbeat information is sent. In addition, heartbeat counters areexchanged only for the nodes 110 that are currently in the healthy loadbalancer node list 320. In at least some embodiments, a load balancernode N can be added to the healthy load balancer node list 320 ofanother load balancer node 110 if node N passes a health check by theload balancer node 110 or if the load balancer node 110 receives agossip message about node N from some other load balancer node 110 onthe list 320.

In at least some embodiments, each load balancer node 110 may maintain asuspect load balancer node list 322, which is a list of load balancernodes whose heartbeat counter (see heartbeat list 326) has not increasedfor a specified interval (referred to as the fail time interval). If aload balancer node E is in the suspect load balancer node list 322 of aload balancer node 110, then the load balancer node 110 will not gossipabout node E. If some other load balancer node 110 on the healthy list320 gossips to the load balancer node 110 about node E with a higherheartbeat counter than the counter for node E in the node 110'sheartbeat list 326, then node E will be moved from the suspect list 322to the healthy list 320. If the node E stays on the load balancer node110's suspect list 322 for a specified interval (referred to as theunhealthy time interval), node E is considered unhealthy by the loadbalancer node 110 and is moved onto an unhealthy node list 324. A node110 on the unhealthy node list 324 (in this example, node G) may bemoved to the healthy node list 320 of a load balancer node 110 upon nodeG passing a health check by the node 110 or upon receiving an updatedheartbeat counter for the node G from another node 110.

In at least some embodiments, each load balancer node 110 may maintain aheartbeat list 326 for all known load balancer nodes 110. For each node,110, this list 326 may include a heartbeat counter and a timestamp thatindicates when the heartbeat counter last changed.

In at least some embodiments, each load balancer node 110 may alsomaintain a heartbeat list for all known server nodes, not shown in FIG.17. This list may be similar to the load balancer node heartbeat list326. In some embodiments, the two lists may be combined. In at leastsome embodiments, the heartbeat information for the server nodes 130 maybe propagated among the load balancer nodes 110, for example accordingto a gossip protocol, along with or in addition to the heartbeatinformation for the load balancer nodes 110.

While FIG. 17 shows four separate lists, it is to be noted that two ormore of the lists may be combined into a single list. For example, insome embodiments, a single list of all nodes 110 may be maintained oneach load balancer node 110, and bit flags or other data structures maybe used to indicate whether each node is currently healthy, suspect, orunhealthy.

Server Node Data

In at least some embodiments, the server nodes 130 and local loadbalancer modules 132 on the nodes 130 do not participate in the gossipprotocol with the load balancer nodes 110. The load balancer nodes 110gossip the heartbeat information about the other load balancer nodes 110obtained by the load balancer node health check method and the heartbeatinformation about the server nodes 130 obtained by the server nodehealth check method only among themselves (specifically, each loadbalancer node 110 gossips only to nodes currently on its healthy loadbalancer node list 320).

However, each server node 130/load balancer module 132 may needinformation about healthy load balancer nodes 110 in the load balancerimplementation so that the server node 130 can determine load balancernodes 110 (specifically, egress nodes) to which the server node 130 canforward outgoing client traffic and determine which load balancer nodesto which connection publishing information is to be sent. In at leastsome embodiments, to provide this information to the server nodes 130,the load balancer nodes 110 may periodically or aperiodically update theserver nodes 130 with information identifying the currently healthy loadbalancer nodes 110 (e.g., healthy load balancer node list 320 in FIG.17). In at least some embodiments, the load balancer nodes 110 that areresponsible for health checking a given server node 130 (see FIG. 15)are responsible for providing the information identifying the currentlyhealthy load balancer nodes to the server 130. For example, referring toFIG. 15, load balancer node 110A may send its healthy load balancer nodelist 320 to server nodes 130A, 130B, 130C, and 130D, load balancer node110B may send its healthy load balancer node list 320 to server nodes130C, 130D, and 130E, and so on.

Handling Load Balancer Node Failures

FIGS. 18A and 18B illustrate handling a load balancer node failure,according to at least some embodiments. FIG. 18A shows an example loadbalancer implementation. There are four load balancer nodes 110A through110D currently in the load balancer implementation. Edge router 104routes incoming packets from clients (not shown) to the load balancernodes 110. In at least some embodiments, the edge router 104 may makethe routing decisions according to a layer 4 per-flow hashed multipathrouting technique, for example an equal-cost multipath (ECMP) routingtechnique. In at least some embodiments, edge router 104 learns aboutthe load balancer nodes 110 that are currently available in the loadbalancer implementation to receive client traffic via load balancer node110 advertisements, for example advertisements via Border GatewayProtocol (BGP) technology sessions initiated by the load balancer nodes110. However, in at least some embodiments, instead of a load balancernode 110 advertising itself to the edge router 104 via a BGP session, atleast one other node 110 in the load balancer implementation takesresponsibility for advertising the node 110 to the edge router 104 viaBGP. For example, in some embodiments as shown in FIG. 18A, the left andright neighbor nodes 110 of a given node 110 advertise the given node110 to the edge router 104. For example, load balancer node 110Aadvertises nodes 110B and 110D, load balancer node 110B advertises nodes110A and 110C, and load balancer node 110C advertises nodes 110B and110D.

As shown in the example of FIG. 18A, each load balancer node 110 alsoperiodically health checks one or more other load balancer nodes 110,for example one or more randomly selected nodes 110, one or moreneighbor nodes 110 as determined by an ordered list of load balancernodes, or one or more neighbor nodes and one or more randomly selectednodes. In addition, each load balancer node 110 may periodically healthcheck at least one server node 130 and also may send its list of healthyload balancer nodes 110 to the server node(s) it health checks. Healthinformation for the load balancer nodes 110 and the server nodes 130 maybe propagated among the nodes 110, for example according to a gossipprotocol.

FIG. 18B illustrates handling the failure of a single load balancer node110 in the example load balancer implementation of FIG. 18A. In thisexample, load balancer node 110B has failed for some reason. Forexample, nodes 110A and 110C may health check node 110B, and both maydetect that node 110B is failing its health checks. Thus, nodes 110A and110C do not increment the heartbeat counter for node 110B. The heartbeatinformation from both nodes 110A and 110B is propagated to the otherhealthy load balancer nodes 110 (in this example, the only other loadbalancer node is node 110D) according to the gossip protocol. As soon asall of the healthy load balancer nodes 110 (in this example, nodes 110A,110C, and 110D) converge on node 110B's failure, one or more of, but notlimited to, the following events may occur. Note that these events donot necessarily occur in this order.

-   -   Nodes 110A and 110C stop advertising node 110B to the edge        router 104. In at least some embodiments, this involves ending        the BGP session that the node 110 established with the edge        router 104 to advertise node 110B. Note that each node 110        establishes a separate BGP session with the edge router 104 for        each other node 110 that it advertises, so ending the BGP        session for node 110B does not affect other nodes 110 that are        advertised. In at least some embodiments, a node 110 ends a BGP        session with the edge router 104 by sending a TCP Close or        similar message for the BGP session to the edge router 104.    -   In response to detecting that node 110B is no longer being        advertised by any of the nodes, edge router 104 stops routing        client data packets to node 110B. The edge router 104 also        adjusts the multipath (e.g., ECMP) hashing to redistribute        packet flows from the clients to the remaining healthy load        balancer nodes 110, specifically to the ingress servers 112 on        the nodes 110. For any packet flow routed to an ingress server        112 for which the ingress server 112 does not have a        client->server mapping, the mapping may be obtained from a flow        tracker node for the client->server connection, or alternatively        a new client->server connection may be established according to        the technique as illustrated in FIGS. 10A through 10G.    -   Nodes 110A and 110C may each opens a BGP session to edge router        104 to advertise each other. Note that, since both nodes 110A        and 110C are advertised to edge router 104 by load balancer node        110D as well as node 110B, the fact that node 110B may stop        advertising nodes 110A and 110B to the edge router 104 when it        fails does not cause edge router 104 to stop routing packets to        these two nodes 110.    -   In at least some embodiments, nodes 110A and 110C may take        responsibility for health checking each other, since they are        now neighbor nodes 110. Note that node 110B, even though        considered unhealthy, may still be randomly health checked by        one or more of the other nodes 110.    -   One or more of the remaining healthy load balancer nodes 110 may        assume responsibility for flow tracking connections formerly        flow tracked by node 110B. For example, node 110C and/or node        110D may take over as primary or secondary flow trackers as        illustrated in FIGS. 11C and 11D for one or more connections for        which node 110B was a primary or secondary flow tracker.    -   One or more of the remaining healthy load balancer nodes 110 may        assume responsibility for health checking the server nodes 130        previously health checked by node 110B. The server nodes 130 are        updated with the healthy load balancer node list (now not        including node 110B) by the remaining load balancer nodes 110.        For example, in FIG. 18B, load balancer node 110A begins health        checking and updating server node 130C, and load balancer node        110C begins health checking and updating server node 130B.    -   On the edge router 104, the BGP sessions from the failed node        110B eventually time out. Alternatively, the edge router 104 may        terminate the BGP sessions upon recognizing that node 110B has        failed.

It is possible that two load balancer nodes 110 can fail at or at closeto the same time. If the two failed load balancer nodes are not adjacentto each other, then the failures are independent and may be handled asseparate single node 110 failures according to the method illustrated inFIG. 18B. However, if the two failed nodes are adjacent to each other(e.g., nodes 110B and 110C in FIG. 18A, then as soon as all of thehealthy load balancer nodes 110 (in this example, nodes 110A and 110D)detect and converge on the failure, one or more of, but not limited to,the following events may occur. Note that these events do notnecessarily occur in this order.

-   -   Node 110A ends the BGP session to edge router 104 for node 110B.    -   Node 110D ends the BGP session to edge router 104 for node 110C.    -   Nodes 110A and 110D start BGP session with edge router 104 to        advertise each other.    -   Nodes 110A and 110D may begin health checking each other. Note        that nodes 110A and 110D may also continue to health check the        failed nodes 110.    -   The remaining healthy nodes 110 update the server nodes 130 with        the healthy load balancer node lists.    -   Traffic may continue to flow from the edge router 104 to node        110B and/or node 110C since these two nodes 110 may continue to        advertise each other to edge router 104. However, these BGP        sessions will eventually timeout, and the edge router 104 will        redistribute the flows to the remaining advertised nodes 110        accordingly.    -   Nodes 110B and 110C may close their BGP sessions with edge        router 104 on which they advertise nodes 110A and 110D,        respectfully, if nodes 110B and 110C think they are still        healthy.

Connection Publishing

Referring again to FIG. 1, in at least some embodiments, the loadbalancer nodes 110 in a load balancer implementation maintain stateinformation for client TCP connections to servers 130. This stateinformation allows the load balancer nodes 110 to route incoming clienttraffic from the edge router 104 to the server nodes 130 responsible forthe TCP connections. The load balancer modules 132 on the server nodes130 maintain lists of active TCP connections to their respective servers134. Connection publishing is a mechanism via which the load balancermodules 132 on the server nodes 130 may publish their lists of activeclient TCP connections to the load balancer nodes 110. In at least someembodiments, the connection publishing packets are formed and publishedto the load balancer nodes 110 by the load modules 132 at a regularinterval, which may be referred to as the connection publishinginterval.

In at least some embodiments, the connection state informationmaintained by the load balancer nodes 110 may be viewed as a form ofcache, and maintaining the state information for a particular connectionmay be viewed as maintaining a lease on the load balancer node 110 forthat connection. Unless the cache entries are renewed, the load balancernodes 110 may not be able to route client data flows to the server nodes130 that are handling the data flows. The connection publishingmechanism periodically renews the caches, and thus the leases, on theload balancer nodes 110 with current connection state information fromthe server nodes 130 to thus keep the TCP packets flowing from theclients 160 to the appropriate server nodes 130. When a client 160 endsa TCP connection to a server 134, the load balancer module 132 on theserver node 130 associated with that connection will drop the connectionfrom its list of active connections and thus will no longer publish theTCP connection through the connection publishing mechanism. Thus, theconnection state information for that connection (the cache entry orentries) on the load balancer nodes 110 associated with that connection(specifically, the ingress server 112 and the primary and secondary flowtrackers 116 for the connection) is no longer renewed, and theconnection is dropped by the load balancer nodes 110. In at least someembodiments, the cache entry or entries for the connection may remain inthe cache on a load balancer node 110 until the memory is required forsome other active connection.

Thus, the connection publishing mechanism periodically or aperiodicallyextends the connection leases on the ingress servers 112 and the primaryand secondary flow trackers 116 to keep the client traffic flowing. Inaddition, the connection publishing mechanism may help recover from atleast some load balancer node 110 failures. When one or more loadbalancer nodes 110 holding state information for a client connectionfails, the active connection information provided to the remaining loadbalancer nodes 110 by connection publishing may in some cases be used torecover the connection.

Using the connection publishing mechanism, the server nodes 130 are theauthoritative sources for the states of the connections between theservers 134 and the clients 160. In addition, closing of connections tothe servers 134 is passively handled by the load balancer modules 132 onthe server nodes 130 and the load balancer nodes 110. Handshaking is notrequired between the server nodes 130 and the load balancer nodes 110.In other words, the load balancer modules 132 do not have to sendmessages to the load balancer nodes 110 to actively inform the nodesthat particular connections have been closed. When a server 134 closes aconnection, the server 134 clears its internal state for the connection.The load balancer module 132 uses the server 134's internal state topopulate the connection publishing packet. Since the connection is nolonger in the server 134's internal state, the connection does not getpublished to the load balancer nodes 110. The lease for the connectionon the load balancer node 110 thus expires, and the load balancer nodes110 passively forget about the connection. Memory in a load balancernode 110's cache that was used for the connection can then be used forother connections as necessary.

In some embodiments, the leases for connections maintained by the loadbalancer nodes 110 may involve time-stamping entries for the connectionsin the cache. When a connection's lease is renewed by a connectionpublishing packet, the timestamp may be updated. If a connection's leaseis not renewed because the connection is no longer being published bythe load balancer module 132 on the server node 130, then the timestampis no longer updated. In at least some embodiments, a lazy garbagecollection method may be used in which the entry for the connection mayremain in the cache until the memory is needed. For example, in at leastsome embodiments, timestamps on cache entries may be compared to a leaserenewal time threshold; if the timestamp for a cache entry is older thanthe threshold, then the entry is stale and may be reused. However, insome embodiments, stale entries may be actively garbage collected.

Connection Publishing Recipients

In at least some embodiments, for each client TCP connection, there arethree load balancer nodes 110 that maintain a connection state—the node110 serving as the ingress server 112, the node 110 serving as theprimary flow tracker 116 and the node serving as the secondary flowtracker 116. For a given TCP flow, the primary and secondary flowtrackers 116 can be determined, for example by a load balancer node 110,by applying a consistent hash function to the TCP flow to find theprimary flow tracker 116 node and its successor node in the consistenthash ring. The load balancer node 110 serving as the ingress server 112for a TCP flow is the node 110 that receives traffic for that flow fromthe edge router 104 based on the edge router 104's internal multipath(e.g., ECMP) hash function. If there is a node 110 failure or addition,the load balancer node 110 serving as the ingress server 112 may changefor many of the active TCP flows; and the load balancer nodes 110serving as flow trackers for at least some active TCP flows may change(see, e.g., FIGS. 11A through 11D). For every TCP flow to the server 132on a server node 130, the load balancer module 132 on that server node130 maintains state information indicating which of the load balancernodes 110 is the ingress server 112 for that TCP flow, since it receivestraffic from that load balancer node 110. However, in at least someembodiments, the load balancer module 132 may not know and may not beable to determine which load balancer nodes 110 are serving as theprimary and secondary flow trackers for a TCP flow, since the loadbalancer module 132 may not know the consistent hash function that isused. In other words, in at least some embodiments, the load balancermodules 132 do not do consistent hashing.

Publishing the Active Connection Information

FIGS. 19A and 19B graphically illustrate a connection publishingtechnique, according to at least some embodiments. FIG. 19A illustratesload balancer (LB) modules publishing active connection information tothe load balancer nodes. In at least some embodiments, each loadbalancer module 132 collects information for each active TCP flow on theserver node 130 and forms a connection publishing packet. Theinformation for a given TCP flow includes information identifying theload balancer node 110 serving as the ingress server 112 for the flow.When a connection publishing packet is ready (e.g., when the connectionpublishing interval has been reached), the load balancer module 132randomly selects a load balancer node 110, for example from the list ofhealthy load balancer nodes 110 that are periodically sent to the servernodes 130 from the load balancer nodes 110 that health-check the servernodes 130 as previously described. The load balancer module 132 thensends the connection publishing packet to the selected node 110. Forexample, in FIG. 19A, load balancer module 132A has sent one connectionpublishing packet to load balancer node 110A, and later sends anotherconnection publishing packet to load balancer node 110B.

FIG. 20 is a high-level flowchart of a connection publishing method thatmay be performed by each load balancer module 132, according to at leastsome embodiments. As indicated at 500, the load balancer (LB) module 132creates a connection publishing entry for every active TCP flow on therespective server node 130. In at least some embodiments, the loadbalancer module 132 retrieves the set of active TCP connections that theserver 134 on the server node 130 handles, for example from/proc/net/tcp on the server node 130. For every active TCP connection,the load balancer module 132 looks up (e.g., in a locally maintainedtable of active connections) the load balancer node 110 that is servingas the ingress server 112 for the TCP flow and creates a connectionpublishing entry that indicates the TCP tuple for the connection (e.g.,a 4-tuple consisting of: the client IP address, client port, server(public) IP address, and server port) and the ingress server 112 for theconnection. Note that each load balancer module 132 maintainsinformation for each active TCP connection indicating the last loadbalancer node 110 from which a packet was received for the connection,and this information may be used by the load balancer module 132 toidentify the ingress node 110 for each active connection.

As indicated at 502, the load balancer module 132 randomly selects aload balancer node 110 to which the connection publishing packet(containing one or more connection publishing entries, with one entryfor each active TCP connection) is to be sent. In at least someembodiments, the load balancer module 110 may be randomly selected whenthe load balancer module 132 determines that the connection publishingpacket is ready to be sent. In at least some embodiments, thisdetermination is made according to a connection publishing interval. Asnon-limiting examples, the connection publishing interval may be 100milliseconds (ms), or one second. In at least some embodiments, the loadbalancer module 110 is selected from a list of healthy load balancernodes 110 that has been previously received from one of the loadbalancer nodes 110. As indicated at 504, the load balancer module thenpublishes the connection publishing packet to the selected load balancernode 110. In at least some embodiments, the connection publishing packetis a stateless packet, for example a UDP packet. In some embodiments,the connection publishing packet may be compressed prior to sending thepackets to the target load balancer node 110. In at least someembodiment, the connection publishing information may be sent to thetarget load balancer node 110 in two or more packets.

As indicated by the arrow returning from element 504 to element 500, theload balancer module 132 may continuously build connection publishingpackets, select random nodes 110, and send the packets to the selectednodes. As noted above, this may be performed according to a connectionpublishing interval so that the load balancer nodes 110 are relativelyregularly refreshed with current active connection information tomaintain the connection leases on the load balancer nodes 110.

In at least some embodiments, since the connection publishing packetsare randomly distributed to the load balancer nodes 110 by the loadbalancer modules, the load balancer nodes 110 that receive theconnection publishing packets are responsible for distributing theactive connection information in the connection publishing packets tothe correct ingress/primary/secondary nodes 110 for the connections.FIG. 19B and FIGS. 21 and 22 illustrate methods for distributing theactive connection information that may be used in at least someembodiments.

FIG. 19B illustrates distributing the active connection informationamong the load balancer nodes 110, according to at least someembodiments. When a load balancer node 110 receives a connectionpublishing packet from a load balancer module 132, the load balancernode 110 may analyze the information for each TCP flow indicated thereinto determine the ingress node and the primary and secondary flow trackernodes for that flow. If the load balancer node 110 is serving in one ofthose roles for a flow, the load balancer node 110 consumes theinformation for the flow (e.g., by updating its cache of stateinformation). In at least some embodiments, the load balancer node 110may also put the information for the flow in packet(s) to be sent to theone or more other nodes 110 that are serving in the other roles for theflow. For the remaining flows indicated by the connection publishingpacket, the load balancer node 110 splits the active connectioninformation into two or more smaller packets and sends each packet toone or more other load balancer nodes 110. For example, in at least someembodiments, a packet containing active connection information for oneor more flows may be sent to the load balancer nodes 110 that areserving as the ingress server 112, primary flow tracker 116A, andsecondary flow tracker 116B for the flow(s).

FIG. 21 is a flowchart of a method for distributing the activeconnection information received in a connection publishing packet totarget load balancer nodes 110, according to at least some embodiments.As indicated at 520, a load balancer node 110 receives a connectionpublishing packet from a load balancer module 132. The load balancermodule 132 generated the packet and selected the load balancer node 110to receive the packet, for example as described above in reference toFIGS. 19A and 20. The connection publishing packet may includeinformation identifying the server node 130 from which the packet wasreceived (e.g., an IP address of the load balancer module 132 on theserver node 130) and a list of entries identifying active TCPconnections (e.g., a 4-tuple consisting of: the client IP address,client port, server (public) IP address, and server port for eachconnection).

In elements 522-530 of FIG. 21, the load balancer module 110 iterativelyprocesses the active TCP connection information indicated in thereceived connection publishing packet. As indicated at 522, the loadbalancer node 110 analyzes the entry for a next TCP flow in the packetto determine the ingress node 110 and the primary and secondary flowtracker nodes 110 for the respective TCP flow. In at least someembodiments, the load balancer node 110 gets the identity of the ingressnode 110 from the connection publishing entry. In at least someembodiments, the primary and secondary flow tracker nodes 110 for theTCP flow may be determined according to the consistent hash function. At524, if the load balancer node 110 is serving in one of the roles forthe TCP flow being examined, then at 526 the load balancer node 110consumes the information for the flow, for example by updating its cacheof state information. As indicated at 528, the load balancer node 110may add the connection publishing entry for the TCP flow to a packetbeing constructed that is to be sent to another load balancer node 110.At 530, if there are more connection publishing entries for flows in theconnection publishing packet, then the method returns to 522 to processthe next entry. Otherwise, the load balancer node sends the newlyconstructed packet(s) each containing a subset of the connectionpublishing entries from the original connection publishing packet totarget load balancer nodes 110 for the packets, as indicated at 532. Inat least some embodiments, the packets sent to the target load balancernodes 110 are stateless packet, for example UDP packets. In someembodiments, the packets may be compressed prior to sending the packetsto the target load balancer nodes 110.

Thus, in at least some embodiments, in elements 522-528 of FIG. 21, theflow tracker node 110 constructs one or more packets (e.g., UDP packets)each to be sent to a particular one of the other nodes 110 according tothe information determined at 522 from the connection publishing entriesin the received connection publishing packet. In at least someembodiments, a packet sent to another node 110 contains entries for TCPflows for which the target node 110 is serving as the ingress node 110,primary flow tracker node 110, or secondary flow tracker node 110. Notethat in some embodiments a given load balancer node 110 may serve asboth the ingress and primary flow tracker node for a TCP flow, or asboth the ingress and secondary flow tracker node for a TCP flow.

FIG. 22 illustrates an alternative method for distributing the activeconnection information received in a connection publishing packet totarget load balancer nodes 110, according to at least some embodiments.As indicated at 550, a load balancer node 110 receives a connectionpublishing packet from a load balancer module 132. In this method, asindicated at 552, a process on the load balancer module 110 analyzes theconnection publishing entries in the packet and splits the receivedpacket into one or more smaller packets accordingly. The load balancermodule 110 does not locally consume the flow information during thisprocess. Once the connection publishing packet has been split into oneor more packets, the packets are then processed as indicated at 554-560.At 554, if the target node 110 for the packet is this load balancer node110, then the load balancer node 110 locally consumes the packet asindicated at 556. Otherwise, the packet is sent to the target loadbalancer node 110. At 560, if there are more packets to be processed,then the method returns to 554. Otherwise, the method is done.

Thus, the load balancer node 110 that receives a connection publishingpacket from a load balancer module 132 may split the connectionpublishing packet into two or more smaller packets that are specific toparticular ones of the other load balancer nodes 110 and distribute thepackets accordingly, while internally consuming flow information for anyTCP flows currently being handled by the load balancer node 110. In themeantime, other load balancer nodes 110 may also be receiving connectionpublishing packets from the load balancer modules 132, splitting theconnection publishing entries in multiple smaller packets, and sendingthe smaller packets to target nodes 110 to thus distribute the activeconnection information among the nodes 110.

Connection Publishing Triggers

In at least some embodiments, a connection publishing may be triggeredon a load balancer module 132 by one or more different events. Aspreviously noted, in some embodiments, a connection publishing packetmay be generated and sent to a randomly selected load balancer node 110according to a connection publishing interval, for example at 100 ms orone second intervals, to renew the leases for the TCP connections on theload balancer nodes 110. In some embodiments, a change in membership ofthe load balancer nodes 110 may trigger an immediate connectionpublishing event. In at least some embodiments, the load balancer module132 may learn about the change from the list of healthy load balancernodes 110 sent from one of the load balancer nodes 110 that healthchecks the respective server node 130. Upon detecting the changeaccording to the list (either a deletion or an addition), the loadbalancer module 132 may generate a connection publishing packet and sendto a load balancer node 110 so that TCP connections affected by thechange may be more quickly recovered by the load balancer nodes 110.

Preventing Packet Loops

Connection publishing packet loops may occur if the load balancer layermembership changes while processing a connection publishing packet. Afirst node 110 may receive a connection publishing packet from a loadbalancer module 132 and send a smaller packet to a second node 110.However, if the membership has changed, the second node 110 maydetermine that the packet should go to the first node 110, and may thusforward the packet to the first node 110. In at least some embodiments,to prevent this loop from happening, different port numbers may be usedfor connection publishing packets received from load balancer modules132 and those received from load balancer nodes 110, and the loadbalancer nodes 110 do not redistribute connection publishing packetsreceived from other load balancer nodes 110.

Connection Publishing Packet Distribution Alternatives

In the connection publishing methods described above, the load balancermodule 132 randomly selects a load balancer node 110 to which aconnection publishing packet is sent. However, in some embodiments,other methods may be used to select a load balancer node 110. Forexample, in some embodiments, the load balancer node 132 may constructone or more connection publishing packets that are each targeted to aparticular ingress node 110 that handles one or more of the active TCPflows, and sent the packet(s) to the target ingress node(s) 110. Theingress node(s) 110 would then redistribute the active connectioninformation to the primary and secondary flow trackers for theconnections. As another example, in some embodiments, instead of sendingthe connection publishing packet to a single, randomly selected node110, each connection publishing packet may be sent by the load balancermodule 132 to two or more of the healthy nodes 110, or to all of thehealthy nodes 110.

Load Balancer Node Architecture

FIG. 23 illustrates example software stack architecture for a loadbalancer node 110 according to at least some embodiments, and is notintended to be limiting. In this example software stack architecture,the load balancer node 110 runs within a single Java™ technology process1102 that uses Java Native Interface (JNI™) 1104 technology to manage alayer of native code that may include load balancer server native code1106 and core packet processing code 1108, for example Intel™ DataplaneDevelopment Kit (DPDK) technology code. The native code may interface totwo network interface controllers (NICs 1114A and 1114B). A first NIC(NIC 1114A) may face “north”; that is, towards the edge router 104. Asecond NIC (NIC 1114B) may face “south”; that is, towards the servernodes 130. In at least some embodiment, NICs 1114A and 1114B may notmaintain TCP stacks. Thus, at least some embodiments may include a thirdNIC 1114C that does support TCP connections so that the load balancernode 110 can communicate with processes via a control plane, and viceversa. Alternatively, in some embodiments, only the first, north-facingNIC 1114A and the second, south-facing NIC 111B may be implemented inthe load balancer node 110, and the second, south-facing NIC 1114B mayimplement a TCP stack via which the load balancer node 110 maycommunicate with processes via the control plane. Load balancer node 110also includes operating system (OS) technology software 1112, e.g. aLinux™ kernel, and a Java Virtual Machine (JVM™) technology software1110 layer on top of OS technology software 1112 and JNI 1104technology.

In at least some embodiments, the load balancer nodes 110 in thedistributed load balancing system may each need to concurrently processmany data flows at high packet rates. In at least some embodiments, toachieve the required level of throughput, the load balancer nodes 110may leverage Intel™ Dataplane Development Kit (DPDK) technology for highperformance packet processing. DPDK technology permits a userspaceprogram to read/write packets directly to and from a network interfacecontroller (NIC) and bypasses the many layers of the Linux kernelnetworking stack (except for the Linus ixgbe base NIC driver). The DPDKapproach to packet processing rejects interrupt handler-based input infavor of dedicated CPU cores that directly poll the NIC hardware in abusy loop. This approach may allow for much higher packet rates, at theexpense of increasing thermal output by continuously running dedicatedCPU cores in a busy loop. DPDK technology may also provide tools forpacket processing including CPU core management, lock-free queues,memory pools, and synchronization primitives. As shown in FIG. 24, inDPDK technology, a dedicated CPU core 600 may be used for eachparticular task, and work is passed from one CPU core 600A to anotherCPU core 600B using non-blocking queues 602.

The DPDK queues 602 may be implemented using fast power-of-two ringbuffers, and may support single and multiple producer/consumer variants.The multiple producer/consumer variants are not truly lock-free, sincethey do contain a compare-and-swap (CAS) loop to synchronize access. Allpacket buffer memory may be pre-allocated in memory pools, so that onlypointers to the buffers are read and written to the queues 602. Thememory pools may be implemented as queues, may be optimized todistribute memory across memory channel and rank, and may supportnon-uniform memory access (NUMA) optimized allocation. In at least someembodiments, the packet buffers may use a method such as an Mbufparadigm that over-allocates enough headroom and tailroom in each packetbuffer to support encapsulate/decapsulate operations that may add/removeouter network layer headers without requiring buffer copies.

In at least some embodiments of the load balancer nodes 110, a corepacket processing architecture may be implemented that leverages DPDKtechnology. Each load balancer node 110 may include at least onemulticore packet processor implemented according to the core packetprocessing architecture. The core packet processing architecture may usea single producer/single consumer paradigm for packet flow through thequeues and cores of the multicore packet processor. In this paradigm,each queue inputs to one and only one core, and each core outputs to oneand only one core for each other core that it feeds packets to. Inaddition, memory used by the cores in the multicore packet processor isnot shared; each core has its own, separate memory region. Thus, thereis no memory or queue sharing between cores, no memory or queuecontention, and no need for memory or queue sharing mechanisms such asrequest for ownership (RFO) or compare-and-swap (CAS). FIGS. 25 and 26illustrate example multicore packet processors implemented according tothe core packet processing architecture.

FIG. 25 illustrates an example multicore packet processor implementedaccording to the core packet processing architecture that leverages DPDKtechnology for processing data flows, according to at least someembodiments. The core packet processing architecture may be implementedas a multicore packet processor according to a single producer/singleconsumer paradigm. In at least some embodiments, as illustrated in FIG.23, the load balancer nodes 110 each have two network interfacecontrollers (NICs)—a north-facing NIC 1114A that faces the bordernetwork/edge router 104 and a south-facing NIC 1114B that faces theproduction network/server nodes 130. In at least some embodiments, theNICs 1114 may be 10 Gpbs NICs. The majority of packets flowing through aload balancer node 110 are received on one of these two NICs (either NIC1114A or 1114B), processed (e.g., encapsulated or decapsulated), andtransmitted out the other NIC (either NIC 1114B or 1114A).

Referring to FIG. 25, in at least some embodiments a load balancer node110 spins up two CPU cores, a receive (RX) core 610 and a transmit (TX)core 630, for each NIC 1114. The load balancer node 110 also spins up anumber of worker cores 620 that process packets for both NICs 1114 inboth directions; in this example four worker cores 620A through 620D areused. The receive cores 610 read batches of incoming packets from theirinput queues as they arrive on the NIC 1114 and distribute the packetsto the worker cores 620 that perform the bulk of the work for eachpacket, with each receive core 610 feeding packets into a respectiveworker input queue 612 for each worker core 620. In at least someembodiment, a receive core 610 may perform a layer 4 “flow-hash”technique on each incoming packet (similar to the per-flow hashedmultipath routing technique that may be used by the edge router 104 aspreviously described) to distribute the packets to the worker cores 620while ensuring that any particular client connection (distinguished byits IP address and port) will be processed by the same worker core 620.This means that each worker core 620 may always see the same subset ofthe packets, and eliminates contention on state data managed by theworker core 620 so that no locks are required. The pointers to thereceived packets may be distributed across the worker queues 622 thatthe worker cores 620 continuously monitor for new input. The workercores 620 are responsible for managing the state (e.g. the assignedserver node 130) for each connection, and may perform UDP encapsulationor decapsulation on the packet before forwarding the packet to one oftheir outbound queues 632. The transmit cores 630 cycle through theworker core 620 outbound queues 632 and write the output packets totheir corresponding NIC 1114 as they appear on the queues 632.

FIG. 26 illustrates another example multicore packet processorimplemented according to the core packet processing architecture thatleverages DPDK technology for processing data flows, according to atleast some embodiments. The core packet processing architecture may beimplemented as a multicore packet processor according to a singleproducer/single consumer paradigm. In at least some embodiments, inaddition to processing the high-throughput client TCP flows, the DPDKcore architecture on a load balancer node 110 may also be used to sendand receive packets on the north- and south-facing NICs 1114 for otherprotocols such as ARP, DHCP, and BGP. In the embodiment shown in FIG.26, a worker core 620A is dedicated to handling the packets for theseother protocols. This worker core 620A may be referred to as a “slow”worker core, since the processing of these packets generally happens ata slower rate than the client TCP flows, while the other worker cores620B-620D that process only the client TCP flows may be referred to asfast worker cores. The receive cores 610A and 610B handling incomingpackets on the north-facing and south-facing NICs 1114, respectively,may identify packets that are to be handled by the slow worker core 620Aand direct the packets to input queues 622 for the slow worker core620A. The slow worker core 620A may also monitor an input queue 622 forpackets generated by Java/JNI, and an output queue 634 for outputpackets to Java/JNI. The slow worker core 620A also outputs to an inputqueue 622 for each of the fast worker cores 620B through 620D so thatthe slow worker core 620A can send packets to each of the fast workercores 620B through 620D, for example connection publishing packets. Theslow worker core 620A also has an outbound queue 632 feeding into eachof transmit cores 630A and 630B.

In at least some embodiments, the third input queue 622 of each fastworker core 620B through 620D is an output queue from the slow workercore 620A. In at least some embodiments, this third input queue 622 may,for example, be used for receiving and processing connection publishingpackets, each containing connection state information, by the fastworker queues 620B through 620D. For at least some of these connectionpublishing packets, there may be no output to the transmit cores 630.Instead, the connection state information in the packets may be consumedby the fast worker core 620, for example by updating the stored statefor one or more packet flows that the respective fast worker core 620maintains. Thus, the output queues from the slow worker core 620A thatinput to the fast worker cores 620B through 620D may provide a pathother than an input queue 622 directly from a receive core 610 forupdating the fast worker cores' stored states.

In at least some embodiments, the multicore packet processors of FIGS.25 and 26 may filter incoming packets and only process and outputpackets that are valid. For example, in at least some embodiments, thereceive cores 610 may filter out packets that are of a protocol notsupported by any of the worker cores 620 and thus not send the packetsto the worker cores 620. In at least some embodiments, the worker cores620, when processing packets, may each first analyze the packets readfrom their respective worker input queues 622 to determine if thepackets are to be accepted for further processing and output to thetransmit cores 630, and may only complete the processing and output ofpackets to the transmit cores 630 that are accepted; the non-acceptedpackets may be discarded. For example, the worker cores 620 may look atthe address information for each packet and only accept packets that aretargeted at valid addresses that are being load-balanced, discarding anyother packets.

Handling Border Gateway Protocol (BGP) Data

In at least some embodiments, packet flows associated with a BGP clientin and out of the core architecture may be handled as follows. Since theNICs 1114A and 1114B are not bound to the Linux kernel, the TCPconnection to the edge router 104 is intercepted by core architecture asillustrated in FIG. 26 and processed by the slow worker core 622A, whichpasses the BGP packets up into Java space via output queue 634. TheseTCP packets are further processed by one or more modules on the loadbalancer node 110 before being delivered to the BGP client, includingprocessing by the Linux kernel to manage the TCP connection andeffectively translate the packets into a TCP stream. This design allowsthe BGP client to be written using standard Java TCP socket libraries.

FIG. 27 illustrates processing of incoming BGP TCP packets by a loadbalancer (LB) node process 650, according to at least some embodiments.A packet from the edge router 104 arrives at the north-facing NIC 640and goes into input queue 640 for the receive core 652. The receive core652 reads the packet from the queue 640, identified the packet as a BGPpacket, and places the packet on an input queue 654 for the slow workercore 656. The slow worker core 656 validates the packet and places it onthe JNI output queue 658. JNI packet receiver 660 reads the packet fromthe queue 658 via JNI, mangles the source/destination addresses, andwrites the packet to a raw socket 644. The Linux kernel 646 receives theraw packet, handles it according to the TCP protocol, and appends thepayload data to the TCP socket InputStream. The data from the packet isthen delivered to the Java TCP socket in the BGP client 662.

FIG. 28 illustrates processing of outgoing BGP TCP packets by a loadbalancer (LB) node process 650, according to at least some embodiments.The BGP client 662 writes data to a Java TCP socket of Linux kernel 646.The Linux kernel 646 handles the data according to the TCP protocol andconverts the data into TCP packet(s). In at least some embodiments, theTCP packet(s) match a 127.x.x.x iptables rule. The TCP packet(s) areplaced on an output queue 648, for example a Netfilter LOCAL_OUT queue.A Java thread of JNI packet receiver 670 monitoring the queue 648 viaJNI receives the TCP packet(s) and marks each NF_STOLEN to make thekernel 646 forget about them. The Java thread mangles thesource/destination addresses and adds the packets(s) to a JNI inputqueue 672 for slow worker core 656 via JNI. The slow worker core 656receives the TCP packet(s) from its JNI input queue 672 and places thepackets on the outbound queue 664 for the north-facing NIC 640 transmitcore 666. Transmit core 666 reads the TCP packets(s) from its inputqueue 664 and writes them to the north-facing NIC 640. The TCP packetsare sent by NIC 640 the edge router 104

Distributed Load Balancer Simulation and Testing

The load balancer described herein is a distributed system that requiresthe interaction of many independent components (e.g., routers, loadbalancer nodes, load balancer modules, etc.). To perform testing of thedistributed components, logic, and protocols, as well as to simulatescenarios such as node failures, message drops, and delays, embodimentsof a test system are described that enable the distributed load balancerto be run in a single process where the interactions can be testedwithout requiring the code to be deployed to multiple hosts in a complexnetwork topology (e.g., a production network). To accomplish this, asoftware mechanism referred to as a message bus is described thatenables multiple load balancer components to be configured and executedin or as a single process; the single process may be executed on asingle host system. The message bus mechanism allows the distributedload balancer system to be tested as a single process, for example on asingle host system, while to the load balancer components (e.g., theload balancer nodes and load balancer modules) it appears that they arerunning on an actual production network.

The message bus provides a framework that allows the distributed loadbalancer to run as a single process. Each of one or more message buslayers in the process simulates a network (e.g., Ethernet) segmentbetween components of the distributed load balancer. The softwarecomponents of the distributed load balancer system do not have to bewritten in a special fashion to allow the components to operate withinthe message bus environment. Instead, the message bus framework providesa component (which may be referred to as a message bus NIC or packetadapter) that intercepts the packets the components of the distributedload balancer system produce, directs the packets into the simulatednetwork provided by a message bus layer instead of into a real physicalnetwork, and delivers the packets to the target components. The messagebus layers do not implement TCP/IP stack(s) for communications betweenthe components. Instead, the message bus layers interface with the hostsystem's operating system (OS) and use the host system's TCP/IP stack.The message bus layers leverage the TCP/IP stack provided by the OS toconvert the TCP streams that the clients and servers expect to and fromthe individual packets that the message bus intercepts and delivers.

In at least some embodiments, to interface with the message bus, loadbalancer components may be provided with at least one message busnetwork interface controller (NIC), each with a valid media accesscontrol (MAC) address, which sends packets to and receives packets fromthe message bus simulated network environment instead of to and from aphysical network. A message bus NIC is a virtual network interfacecontroller that attaches to the message bus instead of to a physicalnetwork. Each load balancer component that needs to communicate throughthe message bus requires a least one message bus NIC. A message bus NICserves as a pipeline exit to the message bus and as a pipeline entranceto the component. Components can instantiate multiple message busnetwork interfaces to each message bus NIC.

A message bus network interface is a mechanism for components to attachto a message bus via a message bus NIC. A message bus network interfacemay be synonymous to an interface configuration (ifconfig) interface inLinux technology, with a difference being that the message bus networkinterface attaches to the message bus instead of to a physical network.A message bus network interface has an IP address, and sits on top of amessage bus NIC. The message bus network interface exposes a packetsource interface, which can be used by the component to receive packetsfrom the message bus, and a packet sink interface that can be used bythe component to send packets into the message bus.

Each load balancer node processes individual network packets that aredelivered and sent through an implementation of the packet source andpacket sink interfaces. When running in the message bus environment,these interfaces are implemented by the message bus network interfacethat adds or removes the layer 2 Ethernet headers (for the load balancernodes that expect this to be performed by the kernel network stack). Ina production environment as shown in FIG. 29, the implementation of thepacket source and packet sink interfaces receive and transmit packets onan actual network interface. In a message bus environment as shown inFIG. 30, the implementation of the packet source and packet sinkinterfaces receive packets from and transmit packets onto a message buslayer or layers.

For the sake of simplicity, a message bus NIC and message bus interfacemay collectively be referred to as a message bus packet adapter, orsimply packet adapter. See, e.g., FIGS. 31 and 32.

FIG. 29 illustrates a load balancing system that includes a distributedload balancer 700 in a production environment, according to at leastsome embodiments. The load balancer 700 has been simplified for thisdescription. The load balancer 700 may connect to clients 742 on anexternal network 740 via a border router 702 of a network installationsuch as a data center that implements the load balancer 700. The loadbalancer 700 includes several types of components—at least one edgerouter 704, two or more load balancer (LB) nodes 710, two or more loadbalancer (LB) modules 732 each implemented on a separate server node(not shown), one or more networking components that form fabric 720 suchas routers or switches, and in at least some embodiments a configurationservice 722. In at least some embodiments, each component of the loadbalancer 700 may be implemented as or on a separate computing device,such as a commodity rack-mounted computing device.

FIG. 30 illustrates a distributed load balancer test system 800 thatincorporates a message bus mechanism that enables multiple distributedload balancing system components to be configured and executed in or asa single process, according to at least some embodiments. In the loadbalancer 700 shown in FIG. 29, each load balancer software component isinstalled and executed on a separate computing device (e.g., the loadbalancer software on the load balancer nodes 710, and the load balancermodules 732 on the server nodes). To enable these load balancer softwarecomponents to execute in a single process, each load balancer softwarecomponent (shown as load balancer (LB) nodes 810 and load balancer (LB)modules 832 in FIG. 30) may include code that abstracts the networkconnectivity of the components so that the packets in and out of theload balancer software component can also be intercepted and routedthrough the message bus mechanism instead of being sent and received ona physical network.

In at least some embodiments, in the distributed load balancer testsystem 800, the message bus mechanism does not implement TCP stack(s)for communications between the components. Instead, the message busmechanism interfaces with the host system's operating system (OS) anduses the host system's TCP stack. In at least some embodiments, themessage bus functionality ties in to the kernel (e.g., the Linux kernel)of the host system's OS below the user layer via IP tables, afunctionality of the kernel. The message bus functionality hooks intothe IP tables at the kernel level, intercepts packets, and sends thepackets up into the message bus process for routing.

As shown by simulated edge router 862 and simulated fabric 864 in FIG.30, the functionality of the physical network components (e.g., the edgerouter 704 and fabric 720 in FIG. 29) may be simulated in software, ascan clients 860, servers 834, and configuration service 866. Note,however, that in at least some embodiments actual rather than simulatedservers 834 may be used in the distributed load balancer test systems800. The message bus layers 850 in FIG. 30 replace the physical networkinfrastructure. Thus, the load balancer software components (loadbalancer nodes 810 and load balancer modules 832) may be run in the loadbalancer test system 800 while unaware that they are not executing in aproduction network environment as shown in FIG. 29.

Some components (for example, simulated routers) may be connected tomore than one message bus layer 850 in order to pass packets to andreceive packets from different message bus layers 850 that simulatenetwork segments.

The message bus mechanism implemented in the message bus layers 850 ofthe distributed load balancing test system 800 simulates the “wire” of anetwork segment. In at least some embodiments, the message bus mechanismdelivers packets to destination components in the distributed loadbalancing test system 800 based on the components' MAC addresses. Thus,each load balancer software component (load balancer nodes 810 and loadbalancer modules 832) provides a MAC address to the message bus layer(s)850 to which it is connected so that the load balancer softwarecomponent can receive packets that are sent to it from other componentsin the distributed load balancing test system 800.

Message Bus Packet Adapters

FIGS. 31 and 32 illustrate message bus packet adapters, according to atleast some embodiments. In at least some embodiments, each load balancer(LB) software component processes individual network packets that aredelivered and sent through an implementation of the PacketSource andPacketSink interfaces. Referring to FIG. 31, when running in thedistributed load balancing test system 800, these interfaces (shown aspacket source interface 862 and packet sink interface 864) may beimplemented by a packet adapter 860 between the message bus layer 850and the load balancer software component 870 that adds or removes thelayer 2 Ethernet headers for the load balancer software components 870that expect this to be performed by the kernel network stack. In theproduction environment as illustrated in FIG. 29, the implementation ofPacketSource and PacketSink for the load balancer software componentsreceives and transmits the packets on actual network interfaces of thephysical devices on which the components are implemented.

Referring to FIG. 31, in at least some embodiments, when a load balancersoftware component 870 transmits a packet, the thread of execution thatcalls a send packet method of packet sink interface 864 traverses achain of functions within the packet adapter 860 and also within themessage bus layer 850 to eventually deliver the packet to thedestination component by adding the packet to that component's inputqueue. In at least some embodiments, when a load balancer softwarecomponent 870 receives a packet, the load balancer software component870 calls a receive packet method of the packet source interface 862 andreads packets from its input queue. In at least some embodiments, themessage bus mechanism does not require any additional threads of its ownto deliver packets.

Message Bus Packet Pipelines

Referring to FIG. 32, in at least some embodiments, the message bus 850side of the packet source interface 862 and packet sink interface 864provides a packet pipeline feature. When a load balancer softwarecomponent 870 sends a packet via packet sink interface 864, the packetdata may traverse a series of stages (packet pipeline 880) beforereaching the message bus layer 850. These stages may modify the packet,drop the packet, duplicate the packet, delay the packet, etc. Once apacket traverses packet pipeline 880 and the message bus layer 850selects a destination component 870, a second series of pipeline stages(packet pipeline 882) associated with the destination component 870 mayalso be traversed before the packet is added to the destinationcomponent 870's input queue.

Example Provider Network Environments

This section describes example provider network environments in whichembodiments of the distributed load balancing methods and apparatus maybe implemented. However, these example provider network environments arenot intended to be limiting.

FIG. 33A illustrates an example provider network environment, accordingto at least some embodiments. A provider network 1900 may provideresource virtualization to clients via one or more virtualizationservices 1910 that allow clients to access, purchase, rent, or otherwiseobtain instances 1912 of virtualized resources, including but notlimited to computation and storage resources, implemented on deviceswithin the provider network or networks in one or more data centers.Private IP addresses 1916 may be associated with the resource instances1912; the private IP addresses are the internal network addresses of theresource instances 1912 on the provider network 1900. In someembodiments, the provider network 1900 may also provide public IPaddresses 1914 and/or public IP address ranges (e.g., Internet Protocolversion 4 (IPv4) or Internet Protocol version 6 (IPv6) addresses) thatclients may obtain from the provider 1900.

Conventionally, the provider network 1900, via the virtualizationservices 1910, may allow a client of the service provider (e.g., aclient that operates client network 1950A) to dynamically associate atleast some public IP addresses 1914 assigned or allocated to the clientwith particular resource instances 1912 assigned to the client. Theprovider network 1900 may also allow the client to remap a public IPaddress 1914, previously mapped to one virtualized computing resourceinstance 1912 allocated to the client, to another virtualized computingresource instance 1912 that is also allocated to the client. Using thevirtualized computing resource instances 1912 and public IP addresses1914 provided by the service provider, a client of the service providersuch as the operator of client network 1950A may, for example, implementclient-specific applications and present the client's applications on anintermediate network 1940, such as the Internet. Other network entities1920 on the intermediate network 1940 may then generate traffic to adestination public IP address 1914 published by the client network1950A; the traffic is routed to the service provider data center, and atthe data center is routed, via a network substrate, to the private IPaddress 1916 of the virtualized computing resource instance 1912currently mapped to the destination public IP address 1914. Similarly,response traffic from the virtualized computing resource instance 1912may be routed via the network substrate back onto the intermediatenetwork 1940 to the source entity 1920.

Private IP addresses, as used herein, refer to the internal networkaddresses of resource instances in a provider network. Private IPaddresses are only routable within the provider network. Network trafficoriginating outside the provider network is not directly routed toprivate IP addresses; instead, the traffic uses public IP addresses thatare mapped to the resource instances. The provider network may includenetwork devices or appliances that provide network address translation(NAT) or similar functionality to perform the mapping from public IPaddresses to private IP addresses and vice versa.

Public IP addresses, as used herein, are Internet routable networkaddresses that are assigned to resource instances, either by the serviceprovider or by the client. Traffic routed to a public IP address istranslated, for example via 1:1 network address translation (NAT), andforwarded to the respective private IP address of a resource instance.

Some public IP addresses may be assigned by the provider networkinfrastructure to particular resource instances; these public IPaddresses may be referred to as standard public IP addresses, or simplystandard IP addresses. In at least some embodiments, the mapping of astandard IP address to a private IP address of a resource instance isthe default launch configuration for all a resource instance types.

At least some public IP addresses may be allocated to or obtained byclients of the provider network 1900; a client may then assign theirallocated public IP addresses to particular resource instances allocatedto the client. These public IP addresses may be referred to as clientpublic IP addresses, or simply client IP addresses. Instead of beingassigned by the provider network 1900 to resource instances as in thecase of standard IP addresses, client IP addresses may be assigned toresource instances by the clients, for example via an API provided bythe service provider. Unlike standard IP addresses, client IP addressesare allocated to client accounts and can be remapped to other resourceinstances by the respective clients as necessary or desired. A client IPaddress is associated with a client's account, not a particular resourceinstance, and the client controls that IP address until the clientchooses to release it. Unlike conventional static IP addresses, clientIP addresses allow the client to mask resource instance or availabilityzone failures by remapping the client's public IP addresses to anyresource instance associated with the client's account. The client IPaddresses, for example, enable a client to engineer around problems withthe client's resource instances or software by remapping client IPaddresses to replacement resource instances.

FIG. 33B illustrates a distributed load balancer implementation in anexample provider network environment as shown in FIG. 33A, according toat least some embodiments. A provider network 1900 may provide a service1910 to clients 1960, for example a virtualized storage service. Theclients 1960 may access the service 1910, for example via one or moreAPIs to the service 1910, to obtain usage of resources (e.g., storageresources or computation resources) implemented on multiple server nodes1990 in a production network portion of the provider network 1900.Server nodes 1990 may each implement a server (not shown), for example aweb server or application server, as well as a local load balancer (LB)module 1992. One or more distributed load balancers 1980 may beimplemented in a load balancer layer between the border network and theproduction network. Border router(s) 1970 may receive packets (e.g., TCPpackets) in packet flows from clients 1960 via an intermediate network1940 such as the Internet, and forward the packets to the edge router(s)of the distributed load balancer(s) 1980 via the border network. Thepackets may be targeted at the public IP address(es) published by theedge router(s) of the distributed load balancer(s) 1980. The edge routerof each distributed load balancer 1980 may distribute the packet flowsamong load balancer nodes of the respective distributed load balancer1980. In at least some embodiments, each load balancer node that servesas an ingress node advertises the same public IP address to the edgerouter, and the edge router distributes the packet flows from theclients 1960 among the ingress servers according to a per-flow hashedmultipath routing technique, for example an equal-cost multipath (ECMP)hashing technique. The load balancer nodes may use the connectionprotocol described herein to determine target server nodes 1990 for thepacket flows and to facilitate connections between the servers and theclients 1960. Once a connection is established, the ingress nodesencapsulate and send packets received for the flows to the target servernodes 1990 on the production network, while the flow tracker nodesmaintain state for the connections. The load balancer modules 1992 onthe server nodes 1990 may make the decisions as to whether therespective servers on the server nodes 1960 accept connections. The loadbalancer modules receive and decapsulate the packets from the ingressnodes, and send the decapsulated packets (e.g., TCP packets) to therespective servers on the server nodes 1990. The load balancer modules1992 may also select load balancer nodes as egress nodes for the packetflows, and encapsulate and send outgoing packets for the flows to theselected egress nodes via the production network. The egress nodes inturn decapsulate the packets and send the decapsulated packets onto theborder network for delivery to the respective clients 1960.

FIG. 34A illustrates an example physical rack implementation of thedistributed load balancer and server nodes according to at least someembodiments, and is not intended to be limiting. In at least someembodiments, various components of the distributed load balancer may beimplemented on or as commodity rack-mounted computing devices. Rack 190may include multiple computing devices each serving as a load balancernode (LB nodes 110A-110F), and multiple computing devices each servingas a server node (server nodes 130A-130L). Rack 190 may also include atleast one edge router 104, one or more rack-mounted networking devices(routers, switches, etc.) that form fabric 120, and one or more othercomponents 180 (other networking devices, patch panels, power supplies,cooling systems, busses, etc.). A network 100 installation such as adata center or centers that implement provider network 1900 of FIGS. 33Aand 33B may include one or more racks 190.

FIG. 34B illustrates another example physical rack implementation of thedistributed load balancer and server nodes according to at least someembodiments, and is not intended to be limiting. FIG. 34B shows the LBnodes 110 and server nodes 130 implemented as slot-mounted computingdevices, for example blade servers, in rack 190.

FIG. 35 illustrates an example networking environment in which one, twoor more distributed load balancers may be implemented in a network, withthe server nodes separately implemented, according to at least someembodiments. In this example, two distributed load balancers 1980A and1980B are shown. The distributed load balancers 1980 each may receivepacket flows from clients 1960 via the border network and perform theload balancing methods described herein to distribute the packet flowsacross multiple server nodes 1990. In some implementations, eachdistributed load balancer 1980 may be a rack implementation similar tothe racks 190 shown in FIGS. 34A and 34B, but without the server nodesinstalled in the load balancer racks. The server nodes 1990 may berack-mounted computing devices such as Blade servers installed in one ormore separate racks within the data center. In some implementations, theserver nodes 1990 may implement two or more different services providedby the provider network, with each service fronted by a different one ormore of the load balancers 1980.

Illustrative System

In at least some embodiments, a server that implements a portion or allof the distributed load balancing methods and apparatus as describedherein may include a general-purpose computer system that includes or isconfigured to access one or more computer-accessible media, such ascomputer system 2000 illustrated in FIG. 36. In the illustratedembodiment, computer system 2000 includes one or more processors 2010coupled to a system memory 2020 via an input/output (I/O) interface2030. Computer system 2000 further includes a network interface 2040coupled to I/O interface 2030.

In various embodiments, computer system 2000 may be a uniprocessorsystem including one processor 2010, or a multiprocessor systemincluding several processors 2010 (e.g., two, four, eight, or anothersuitable number). Processors 2010 may be any suitable processors capableof executing instructions. For example, in various embodiments,processors 2010 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitableISA. In multiprocessor systems, each of processors 2010 may commonly,but not necessarily, implement the same ISA.

System memory 2020 may be configured to store instructions and dataaccessible by processor(s) 2010. In various embodiments, system memory2020 may be implemented using any suitable memory technology, such asstatic random access memory (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash-type memory, or any other type of memory. In theillustrated embodiment, program instructions and data implementing oneor more desired functions, such as those methods, techniques, and datadescribed above for the distributed load balancing methods andapparatus, are shown stored within system memory 2020 as code 2024 anddata 2026.

In one embodiment, I/O interface 2030 may be configured to coordinateI/O traffic between processor 2010, system memory 2020, and anyperipheral devices in the device, including network interface 2040 orother peripheral interfaces. In some embodiments, I/O interface 2030 mayperform any necessary protocol, timing or other data transformations toconvert data signals from one component (e.g., system memory 2020) intoa format suitable for use by another component (e.g., processor 2010).In some embodiments, I/O interface 2030 may include support for devicesattached through various types of peripheral buses, such as a variant ofthe Peripheral Component Interconnect (PCI) bus standard or theUniversal Serial Bus (USB) standard, for example. In some embodiments,the function of I/O interface 2030 may be split into two or moreseparate components, such as a north bridge and a south bridge, forexample. Also, in some embodiments some or all of the functionality ofI/O interface 2030, such as an interface to system memory 2020, may beincorporated directly into processor 2010.

Network interface 2040 may be configured to allow data to be exchangedbetween computer system 2000 and other devices 2060 attached to anetwork or networks 2050, such as other computer systems or devices asillustrated in FIGS. 1 through 35, for example. In various embodiments,network interface 2040 may support communication via any suitable wiredor wireless general data networks, such as types of Ethernet network,for example. Additionally, network interface 2040 may supportcommunication via telecommunications/telephony networks such as analogvoice networks or digital fiber communications networks, via storagearea networks such as Fibre Channel SANs, or via any other suitable typeof network and/or protocol.

In some embodiments, system memory 2020 may be one embodiment of acomputer-accessible medium configured to store program instructions anddata as described above for FIGS. 1 through 35 for implementingembodiments of a distributed load balancing system. However, in otherembodiments, program instructions and/or data may be received, sent orstored upon different types of computer-accessible media. Generallyspeaking, a computer-accessible medium may include non-transitorystorage media or memory media such as magnetic or optical media, e.g.,disk or DVD/CD coupled to computer system 2000 via I/O interface 2030. Anon-transitory computer-accessible storage medium may also include anyvolatile or non-volatile media such as RAM (e.g. SDRAM, DDR SDRAM,RDRAM, SRAM, etc.), ROM, etc, that may be included in some embodimentsof computer system 2000 as system memory 2020 or another type of memory.Further, a computer-accessible medium may include transmission media orsignals such as electrical, electromagnetic, or digital signals,conveyed via a communication medium such as a network and/or a wirelesslink, such as may be implemented via network interface 2040.

CONCLUSION

Various embodiments may further include receiving, sending or storinginstructions and/or data implemented in accordance with the foregoingdescription upon a computer-accessible medium. Generally speaking, acomputer-accessible medium may include storage media or memory mediasuch as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile ornon-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.),ROM, etc, as well as transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as network and/or a wireless link.

The various methods as illustrated in the Figures and described hereinrepresent exemplary embodiments of methods. The methods may beimplemented in software, hardware, or a combination thereof. The orderof method may be changed, and various elements may be added, reordered,combined, omitted, modified, etc.

Various modifications and changes may be made as would be obvious to aperson skilled in the art having the benefit of this disclosure. It isintended to embrace all such modifications and changes and, accordingly,the above description to be regarded in an illustrative rather than arestrictive sense.

What is claimed is:
 1. A distributed load balancer system, comprising: aplurality of load balancer nodes; and a plurality of server nodes eachcomprising a server and a load balancer module; wherein the plurality ofload balancer nodes is configured to distribute packet flows from one ormore clients among the plurality of server nodes, wherein, to distributethe packet flows among the plurality of server nodes, the plurality ofload balancer nodes is configured to: select server nodes from among theplurality of server nodes to receive connection requests for the packetflows from the clients; and send the connection requests to the selectedserver nodes; wherein the load balancer module on each server node isconfigured to: receive a connection request for a packet flow from oneof the plurality of load balancer nodes; determine if the connection isto be accepted by the server on the server node; if the server cannotaccept the connection, reject the connection request; and if the servercan accept the connection, cooperate with the plurality of load balancernodes to establish a connection for the packet flow between therespective client and the respective server.
 2. The distributed loadbalancer system as recited in claim 1, further comprising a routerconfigured to distribute the packet flows from the one or more clientsamong the plurality of load balancer nodes according to a hashedmultipath routing technique.
 3. The distributed load balancer system asrecited in claim 1, wherein, to determine if the connection is to beaccepted by the server on the server node, the load balancer module isconfigured to analyze one or more current resource usage metrics of theserver on the server node to determine if the server can accept theconnection, wherein the one or more current resource usage metricsinclude one or more of CPU utilization, bandwidth consumption, serverlatency, and number of established connections.
 4. The distributed loadbalancer system as recited in claim 1, wherein the plurality of loadbalancer nodes is further configured to select the server nodes fromamong the plurality of server nodes to receive the connection requestsaccording to a random selection technique.
 5. The distributed loadbalancer system as recited in claim 1, wherein the plurality of loadbalancer nodes is further configured to select another server node fromamong the plurality of server nodes to receive a rejected connectionrequests and send the connection request to the other server node. 6.The distributed load balancer system as recited in claim 1, wherein eachpacket flow is a Transmission Control Protocol (TCP) packet flow, andwherein each connection established between a client and a server is aTCP connection.
 7. The distributed load balancer system as recited inclaim 1, wherein each connection established between a client and aserver originates at the client, passes through one or more of theplurality of load balancer nodes, and is terminated by the server.
 8. Amethod, comprising: performing, by one or more of a plurality of loadbalancer nodes: receiving a packet in a packet flow for a client; andsending a connection request for the packet flow to a server nodeselected from among a plurality of server nodes; performing, by theselected server node: determining if a server on the server node can orcannot accept the connection; rejecting the connection request upondetermining that the server cannot accept the connection; and acceptingthe connection request upon determining that the server can accept theconnection.
 9. The method as recited in claim 8, wherein said acceptingthe connection request comprises the selected server node cooperatingwith the one or more load balancer nodes to establish a connectionbetween the respective client and the respective server for the packetflow.
 10. The method as recited in claim 9, wherein the packet flow is aTransmission Control Protocol (TCP) packet flow, and wherein theconnection established between the client and the server is a TCPconnection.
 11. The method as recited in claim 9, wherein theestablished connection originates at the client, passes through one ofthe plurality of load balancer nodes, and is terminated by the server.12. The method as recited in claim 8, wherein the packet is receivedfrom a router that distributes packet flows from one or more clientsamong the plurality of load balancer nodes according to a hashedmultipath routing technique.
 13. The method as recited in claim 8,wherein said determining if a server on the server node can or cannotaccept the connection comprises analyzing one or more current resourceusage metrics of the server to determine if the server can accept theconnection.
 14. The method as recited in claim 8, further comprising theone or more load balancer nodes selecting the server node from among theplurality of server nodes according to a random selection technique. 15.The method as recited in claim 8, further comprising, if the selectedserver node rejects the connection request, the one or more loadbalancer nodes sending the connection request to another server nodeselected from among the plurality of server nodes.
 16. A non-transitorycomputer-accessible storage medium storing program instructionscomputer-executable to implement a load balancer module on each of aplurality of server nodes, each load balancer module operable to:receive, from one of a plurality of load balancer nodes, a connectionrequest for a packet flow from a client; determine if a server on theserver node can or cannot accept the connection; reject the connectionrequest upon determining that the server cannot accept the connection;and communicate with the load balancer node and with the server toestablish a connection between the client and the server for the packetflow upon determining that the server can accept the connection.
 17. Thenon-transitory computer-accessible storage medium as recited in claim16, wherein, to determine if a server on the server node can or cannotaccept the connection, the load balancer module is operable to analyzeone or more current resource usage metrics of the server to determine ifthe server can accept the connection.
 18. The non-transitorycomputer-accessible storage medium as recited in claim 17, wherein theone or more current resource usage metrics include one or more of CPUutilization, bandwidth consumption, server latency, and number ofestablished connections.
 19. The non-transitory computer-accessiblestorage medium as recited in claim 16, wherein the program instructionsare further computer-executable to implement the load balancer noderandomly selecting the server node from among the plurality of servernodes to receive the connection request.
 20. The non-transitorycomputer-accessible storage medium as recited in claim 16, wherein, toreject the connection request, the load balancer module is operable todecrement a time to live (TTL) counter in the connection request andreturn the connection request to the load balancer node, wherein theprogram instructions are further computer-executable to implement theload balancer node: checking the TTL counter in the returned connectionrequest; if the TTL counter is above a threshold, selecting anotherserver node from among the plurality of server nodes to receive theconnection request; and if the TTL counter is at or below the threshold,dropping the connection request.